UNCLASSIFIED 


DEFENSE  DOCUMENTATION  CENTER 

FOR 

SCIENTIFIC  AND  TECHNICAL  INFORMATION 

CAMERON  STATION  AIEXANDRIA.  VIRGINIA 


REPRODUCED  FROM 
BEST  AVAILABLE  COPY 


UNCLASSIFIED 


DISCLAIMER  NOTICE 


THIS  DOCUMENT  IS  BEST  QUALITY 
PRACTICABLE.  THE  COPY  FURNISHED 
TO  DTIC  CONTAINED  A  SIGNIFICANT 
NUMBER  OF  PAGES  WHICH  DO  NOT 
REPRODUCE  LEGIBLY. 


90T1CS:  When  government  or  other  druvi'  js,  sptcl- 
fl cations  or  other  data  are  ueed  for  ear  purpo-e 
other  than  In  connection  with  a  deftnit*  ly  Rated 
government  procurement  operation,  the  U.  S. 
Government  thereby  Incurs  no  responsible  lty,  wr  any 
obligation  vhatr sever;  and  the  fact  tla;  the  Cc/ern- 
ment  nay  have  formulated,  furnished,  or  In  anr  way 
supplied  the  said  drawings,  specificati  <ae,  or  tier 
data  la  not  to  be  regarded  by  implication  or  o'..*?* 
vise  as  in  any  manner  licensing  the  holder  or  wy 
other  person  or  corporation,  or  conveying  say  rights 
or  permission  to  manufacture,  use  or  sell  any 
patented  Invention  that  may  in  any  way  be  related 
thereto. 


i 


:i2r  .tswal 


,  v 


i 


v 

4  V 


V 


r  !•' 
v 

-4  1 
'  ,  '< 

>t  44 
•i  fi 

i  V 


i 

v 


r  »: 

.V 


n 


*  vV.  I  *  r 


‘  *  .  VT  *  *  4 


*  *  ) 1  v  if  V  * 

ARLINGTON  H  ALL  ST/,  .rot 
A  ILINGTON  12  VIRGINIA 


*  >  • 

I  •, 


1 


•'  i  s'* 

c  <  • 


I  1 


<i 

I 


/  ;  -fi  t 

'  •  'll-', 

'  '’  '‘  I 


*  ^ 
i ; 


i 

•t 


C  W 


1 


v  tumiYr  or  other  dra'  was,  specif*  r  flf  -  *  OTH^RitA.T*  „ 

.  .  OTHER  THAN  IN  C'.OWNCCTION  Wv  \J>*  * CTELY  RELATED 

4  „**  OPERATION,  THU  If.  &  OOVSRNU*  ?  »  '.REBY  INCURS 

S*  .  **i  AIT  OHURiATfOW  Wlfi.\T50RVER;  ANC  t  f  !T  THAT  THE 

>o  ;*  *  i  ?C*U4VLAr’*  \  PlIXKIliKD.  OH  IN  ANV  »i  UPPLDKDTHE 

S*  •  >-  v"  «ATIU:'  'SEXn./TAtSNOT  TO  ft;  ».<  APED  BY 

4  :.v?  UKHE'4  UCEXSTNO  T, .  *'  SR  OR  ANY  OTHER 

•'  «  .  •  , s.*  w .  v  im  i  iuw  >wwm  no  r.r»i  1  >  TO  MANUFACTURE 


'"V  -* 

<S 


:  » 

» 

/• 


V  <r 

>.  U 


•  <5*  M  yt  ANY  WORTS  OR  F*»;s  <t>  3  i  TO  MANUFACTURE, 

»  •*..»'»  -  *  i  '»**.  . JW  HAT  MAT  IN  ANY  W i  \  *  SLATED  THERETO. 

-»%  -v ,  1WW*  *  — — —  mmmmm  mm 


1  1 


I  4 

I 

i 


s4 


M 


#  -  '  4? ;/ 

‘  ’  |  *'  y  i<i'  E  Ur  ^^I5f 

r  : » i  , /,.%  ^  ,1  V  TiTx* 


4 


r 


TECHNICAL  REPORT  3*5 

MARCH  X  l*M 


fin  co*y 


■•turn  »• 

astia 

ARLINGTON  MAll  STATION 

arunoton  »J.  vi»o,mu 

A*f»*i  TlSSj 


MASSACHUSETTS  INSTITUTE  OP  TECHNOLOGY 
RESEARCH  LABORATORY  OP  ELECTRONICS 
CAMMI06C,  MASSACHUSETTS 


ASTIA 


r 


REPRODUCED  FROM 
BEST  AVMLABIECOPy 


MASSACHUSETTS  INSTITUTE  OF  TECHNOLOGY 
RESEARCH  LABORATORY  OF  ELECTRONICS 


Technical  "V  port  545 


March  5.  195* 


THEORY  OF  THE  ANALYSIS  OF  NONLINEAR  SYSTEMS 

Martin  B.  Brilliant 


This  report  is  baaed  on  a  thesis  submitted  to  the  Department  of 
Electrical  Engineering,  January  !  5,  l<*58,  in  partial  ful¬ 

fillment  o'  the  requirements  for  the  degree  of  Doctor  oi  science. 


Abstract 

A  theory  of  lhe  analysis  of  nonlinear  systems  la  developed.  The  central  problem  la 
the  mathematical  rcocesentation  of  the  dependence  of  the  value  or  the  output  of  such 
systems  on  the  present  and  past  of  the  input.  It  is  shown  that  these  systems  can  be  con¬ 
sidered  as  generalized  fundi  -.a,  and  that  many  mathematical  methods  used  for  the 
representation  of  functions  of  a  real  variable,  particularly  tableB  of  values,  polynomials, 
and  expansions  in  series  of  orthogonal  functions,  can  be  used  in  generalized  form  for 
nonlinear  ayatems. 

The  discussion  is  restricted  to  time -invariant  systems  with  boundrd  inputs.  A  deft, 
nitlon  of  s  continuous  system  is  given,  and  it  is  shown  that  any  continuous  system  can  he 
approximately  represented,  with  the  error  as  small  as  may  be  required,  by  the  methods 
mentioned  above,  rtougbly  described,  a  continuous  system  Is  one  that  is  relatively 
insensitive  to  small  cnanges  in  the  input,  to  rapid  fiuctuaiions  (high  frequencies)  in  the 
tnput,  and  to  the  remote  past  of  the  input, 

A  system  is  called  an  analytic  System  if  it  can  be  exactly  represented  by  a  certain 
formula  that  is  a  power-series  generalization  of  the  convolution  integral.  This  formula 
can  represent  not  only  continuous  systems  but  also  no-memory  nonlinear  systems. 
Methods  sre  derived  for  calculating,  m  analytic  form,  the  results  of  inversion,  addition, 
multiplication,  cascade  combination,  and  simple  feedback  connection  of  analytic  systems. 
The  resulting  tenet  it  proved  to  be  convergent  under  certain  conditions,  and  bounds  are 
derived  for  the  radius  of  convergence,  the  output,  and  the  error  incurred  by  using  only 
the  first  few  terms.  Methods  are  suggested  for  the  experimental  determination  of  ana¬ 
lytic  representations  for  given  systems. 


I.  INTRODUCTION 
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!  1. 1  NONLINEAR  SYSTEMS 

At  the  present  time  the  most  useful  method*  for  mathematical  tniljrtu  and  design  rf 
electrical  systems  are  baaed  on  the  theory  of  linear  systems.  The  technique*  of  analysts 
and  design  of  linear  systems  have  been  well  developed,  and  they  are  used  not  only  for 
perfectly  linear  systems  but  also  for  almoct  linear  system*. 

Many  communication  and  control  devices  are  not  dearly  linear.  Sometimes  nonlin¬ 
earity  tc  essential  to  the  operation  of  a  device,  sometimes  it  is  undesirable  but  unavoid¬ 
able,  and  sometime*  a  nonlmear  component,  although  it  ts  not  essential,  may  give  better 
reaulta  than  any  linear  component  that  might  be  used  m  its  place,  some  times  nonlinear¬ 
ity  i*  avoided,  not  because  it  would  have  an  undeatred  effect  in  practice,  but  simply 
because  i»a  effect  cannot  be  c. imputed.  There  has  therefore  been  an  increasing  effort  to 
develop  method-  •'f  analysis  and  design  for  nonlinear  devices. 

It  is  appropriate  to  note  here  the  relation  between  linear  and  nonlinear  systems.  A 
nonlinear  system  can  be  s’morl  linear,  but  there  la  no  such  thing  aa  a  linear  system  that 
1a  almost  nonlinear.  Th.  linear  case  is  a  limiting  caae  of  nonlinearity,  and  it  ia  an 
especially  simple,  not  a  especially  difficult,  limiting  esse. 

We  should  expect,  therefore,  that  any  theory  or  technique  that  is  adequate  for  general 
nonlinear  systems  must  Li  equally  adequate  for  linear  systema.  The  word  "nonlinear*  is 
appropriate  only  to  special  technique  •  a  general  theory,  applicable  to  both  linear  end  non¬ 
linear  systems,  sho  uld  not  be  called  “nonlinear,*  but  ‘general.*  However,  the  designation 
•nonlinear*  will  >*  used  In  this  report  to  Indicate  the  breadth  of  the  theory,  with  the 
understanding  ths*  it  is  not  to  be  Interpreted  ittersdly  as  excluding  the  special  linear  case. 

1.2  HISTORICAL  BACKGROUND 

V  '«■*  effort  to  develop  techniques  of  nonlinear  system  analysis  has  been  prims, 

rily  associated  with  a  number  of  Russian  schools.  In  'his  connection  Poincsrd,  although 
he  was  not  a  Russian,  m.ist  be  mentioned,  as  well  a*  UtapounofT,  Andronov  and  Chaikin, 
Kryloll  and  Bogoliuboff.  A  great  deal  of  this  work  was  summarised  by  Minorsky  fl)  and 
published  in  1*447.  This  esrlis:  research  was  directed  principally  toward  the  solution  of 
nonlinear  differential  equations  and  the  micaUgatlon  of  the  properties  ot  their  solutions. 
Fruitful  as  this  work  was,  its  scope  is  limited,  i-.vo  it  has  played  no  part  in  the  author's 
research. 

The  author's  research  la  based  on  the  representation  of  nonlinear  systems  by 
expressing  the  output  directly  In  terms  of  the  input.  The  root*  of  this  approach  might 
be  historically  :-»c*  d  to  Volterro  (2),  who  Included  a  theory  of  analytic  functionals  in  Ms 
•Lecons  *•>•  ier  "bons  d*  lignes*  in  1911.  In  1*21,  Wiener  (1)  brought  the  theory  of 
Brown's  <  motion  to  bear  on  n  >  p  rPOem  of  defiring  an  integral  over  s  space  of  functions, 
v.nd  included  a  discussion  of  Lbs  average  of  »i<  analytic  functional.  In  1942,  Wiener 
brought  Brownian  motion  and  analytic  functionals  logrvher  again  (4).  The  later  paper 
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contains  the  firat  use.  In  the  representation  of  nonlinear  systems,  of  'he  formula  that 
forme  the  bnata  of  Section  IV  of  this  report,  and,  th  *'ct,  it  aeema  to  be  the  firat  attempt 
at  a  general  formula  for  the  repreaen'auon  of  nor  ear  ayatema.  Some  other  work  along 
tho  name  ltnea  waa  done  more  recently  Sy  IXeharu  (5)  in  1941.  and  by  Deutach  IS)  in  J 955. 

In  recent  yeara  Wiener  developed  a  general  repreaenlation  method  for  nonlinear  aye. 
tern*  that  ia  baaed  on  the  properties  of  Brownian  motion,  but  doer  not  employ  the  formula 
that  he  used  in  1942.  This  theory  differs  from,  the  1942  report  in  that  it  attacks  the  gen. 
eral  nonlinear  problem  rather  titan  the  specific  problem  of  noise  in  a  particular  class  of 
systems.  The  method  has  been  prcsmled  in  unpublished  lectures  and  described,  although 
not  in  its  most  recent  form,  by  Boo  ion  (?)  and  by  Bose  It,  9).  Theoretical  approaches 
related  to  this  method  have  been  developed  ny  Singleton  (10)  and  by  Bose  (9).  The  repre. 
Mentation  formula  developed  by  £adrh  (11)  is  similar  in  its  basic  orientation. 

i.J  SYSTEMS  AND  FUNCTIONS 

One  of  the  central  problems  ir  the  ar.ilyrvs  of  nonlinear  ayatema  is  the  finding  of  a 
good  repreaenlation  formula .  ouch  .  formula  must  be  able  to  represent,  either  exactly 
or  w.'*n  arbitrarily  a.-.iai.  error,  a  Large  class  of  systems;  and  it  must  also  be  convenient 
f<v  use  in  i  'Inula. ,m..  •r.volv*1'  systems. 

There  Is,  however,  u  representation  problem  In  a  more  fundamental  sense.  It  is 
necessary  to  :  uie  .aea  of  a  nonlinear  system  to  more  fundamental  concepts.  This 

’tec  on  abr.tr  ct  representation,  •  hose  generality  is  not  limited  by  any  concession 
fc,  mpuutional  convenience.  W  ;n  rtuch  a  representation  at  hand,  representation  for. 
mulaa  designed  for  computational  needs  can  be  more  easily  apprehended. 

This  abstract  representation  is  found  in  the  general  concept  of  a  function.  A  function, 
abstractly  defined,  is  a  relation  between  two  sets  of  objects,  called  the  domain  and  the 
-  ange  of  the  iunclion.  which  assigns  to  every  object  in  the  domain  a  corresponding  object 
in  tne  rang*,  with  every  object  in  the  range  assigned  tc  at  least  one  object  in  the  domain, 
it  may  be  uaiJ  that  a  function  is  any  relation  of  tha  form  *plug  in  x,  out  cornea  y*;  the 
set  of  a:  ;  thu<  can  1  «  plugged  in  is  the  domain,  and  the  ret  of  all  y  that  can  come  out 
Is  the  range. 

This  definition  implies  no  restriction  on  the  nature  of  the  objects  x  and  y.  We  may 
have,  lor  example,  an  amplifier  chassis  with  an  empty  tube  socket;  every  tube  that  can 
be  inserted  in  the  socket  will  give  us  a  different  amplifier.  Therefore,  we  have  a  func. 
tion;  the  domain  of  this  function  is  the  set  of  all  tubes  that  can  be  Inserted  in  the  socket, 
and  the  range  ie  the  set  of  tl'.  impUficrs  that  can  thus  be  obtained. 

We  are  most  familiar  with  functions  whose  domain  snd  range  are  sets  of  real  num¬ 
bers.  Such  (unctions  arc  called  "real-valued  functions  of  s  real  vsriahle*;  for  conven¬ 
ience,  we  shall  call  them  'real  functions.*  In  general,  any  function  whose  range  la  a 
set  or  real  numbers  is  called  a  "real. valued  function.3  A  real  function  ia  usually  repre. 
sented  by  a  letter,  such  as  (,  The  equation  y  *  fix)  means  that  y  is  the  element  of  the 
range  which  f  assigns  to  the  element  x  of  the  domain.  Note  that  f(x)  is  not  a  function. 
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but  a  value  of  the  function  f;  that  1a,  an  element  of  fee  ruft. 

A  nonlinear  system  with  one  input  and  one  output  is  a  function  ice:  ."_ng  *'  -tea  deft- 
nition.  Far  every  input  in  th»  set  of  tnputa  that  the  system  ta  d<r*tgnro  to  arcvpt.  fee 
system  produces  a  corresponding  output.  These  inputs  and  outputs  cat.  "urselm  he 
reprerented  by  functions.  If  we  assume  that  the  inputs  and  outputs  are  <  nine  upwls, 
they  can  be  described  by  real  functions.  To  every  read  number  t  there  \-.  imped  s 
corresponding  real  number  fit)  that  represents  the  \a-we  of  the  input  or  c  -  -a*  at  time  t. 

A  nonlinear  aystem  can  therefore  be  represented  by  a  function  whose  dunlin  and  range 
are  sets  of  real  iVnctlona.  Although  such  a  function  a  conventionally  called  at  ‘operator* 
or  a  *tran  >fc  rmaticn,*  it  will  be  referred  to  in  this  report  as  a  *hyperf  — vcuon*  to  empha¬ 
size  the  fa.  i  that  it  is  s  function.  A  hyperfuncuoo  (or  the  system  it  repr-seats)  will  be 
denoted  by  a  capital  script*  letter;  the  equation  g  a  H(Q  states  that  g  is  the  ml  function 
that  represents  the  output  of  the  aystem  U  when  tbe  utput  is  the  signal  represented  by  the 
real  function  f. 

Most  of  the  discussion  in  the  following  sections  deals  specifically  with  use-invariant 
systems,  Such  systems  can  be  represented  by  a  kind  of  function  that  is  simpler  than  a 
hyperfunction  —  a  function  whose  domain  is  a  set  of  real  functions  and  whose  range  is  a 
set  of  real  numbers.  Such  functions  are  conventionally  called  'functionals.* 

The  argument  will  be  simpler  if  we  consider  only  pbyan-aUy  realisable  systems,  that 
is.  systems  in  which  the  value  of  the  output  at  any  lime  does  not  depend  on  future  valuer 
of  the  input,  if  the  system  H  is  physically  realisable  and  time -invariant,  then  me  output 
at  a  particular  time  t  can  be  determined  without  knowing  either  the  value  of  t  or  the 
time  at  which  each  value  of  the  input  occurred;  it  is  sufficient  to  specify,  for  every  non- 
negative  number  t,  what  the  value  of  the  input  was  *  seconds  ago.  This  input  feta  esn 
be  expressed  by  the  real  function  u,  u(rl  a  fit— v)  for  v  »  0,  where  f  represents  the  input 
in  the  us.al  form.  To  each  function  u  there  corresponds  a  unique  real  number  fe(u),  wi*h 
the  property  that  the  value  of  the  output  of  the  gys*em  is  h(u)  whenever  the  pass  of  the 
input  ia  represented  sy  u.  The  function  h  is  a  functional  according  to  the  definition  given. 
For  a  specified  Input  f,  the  function  u  will  be  different  for  different  l  and  will  be  desig¬ 
nated  as  Uj  if  t  is  to  be  specified;  as  t  changes,  u(  changes,  and  the  value  of  the  output 
changes  with  it.  if  the  system  U  is  not  physically  realisable,  but  is  still  time  -in  variant, 
the  only  change  that  is  necesaary  u»  this  argument  is  to  define  u(t)  for  all  v. -negative  as 
well  ns  positive. 

For  the  most  part,  we  shaH  consider  systems  for  bounded  inpu.s  only.  A  real  func. 
tion  f,  representing  an  input,  will  be  called  boundedlRI  if  |f(t)j  <  R  for  all  t.  The  set 
of  all  real  functions  u,  u(t)  defined  for  i  >0,  that  are  bounded(R),  will  be  called  PBI(R). 
(PBI  stands  for  Past  of  Bounded  Input.)  All  these  real  functions  will  be  assumed  to  be 

*  Editor's  rr.te:  With  the  pernuenon  of  the  author,  the  script  let;*  ~9  originally  used 
(i.e.,  Jt'.  A,  X,  etc.)  have  been  replaced  with  the  corresponding  typed  letter  and  iden¬ 
tified  by  an  underline. 
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Uebesgue  measurable;  in  practice  this  U  no  restriction,  since  some  tricky  mathematical 
work  is  required  to  prove  the  existence  of  function*  that  ere  not  Lebesgue  measurable. 
Such  "improper  function*1  a*  impulses  or  tnfiMte-bandwidth  white  nolo*  are  not  realty 
function*,  arid  thu*  their  measurability  i*  questionable,  but  they  are  excluded  from  con¬ 
sideration  a*  possible  inputs  on  the  ground  that  they  are  not  bounded. 

We  shall  always  consider  two  real  functions  f  and  g  to  toe  equivalent  if 

J*  (f(*>  -  r(x>}  dx  •  o  to 

for  all  real  number  a  a  and  b,  since  two  such  functions  are  indistinguishable  by  any 
physical  measurement  proceas. 

1.4  REPRESENTATION  OF  FUNCTIONS 

The  central  problem  of  computationally  convenient  representation  can  now  be  treated 
with  some  perspective.  We  have  to  find  convenient  representations  for  certain  kinds  of 
functions,  namely,  functionals  and  hyperfunctions. 

Suitable  methods  can  l>e  derived  by  generalizing  the  faruliar  methods  used  for  the 
representation  of  real  functions.  These  Include:  (a)  miscellaneous  designations  for 
special  functions,  e.g.,  algebraic,  trigonometric;  (b)  implicit  functions;  fc;  tables  of 
values;  (d)  polynomials.  Including  power  series;  and  fe)  expansions  in  serle.->  f  orthog¬ 
onal  functions.  The  last  three  are  methods  of  approximate  representation,  or  represen¬ 
tation  as  a  limit  of  successive  approximation*.  All  the  methods  mentioned  In  section  1.2 
are  particular  forms  of  generalizations  of  these  method*. 

Several  classes  of  specially  designated  systems,  that  is,  method  (a)  of  the  preceding 
paragraph,  are  already  well  known.  Perhaps  the  moat  important  is  the  class  of  linear 
systems,  whose  special  representation  by  means  of  the  convolution  Integral  has  been 
found  particularly  convenient.  No-mcmory  .y  stems  (the  value  of  whose  output  at  any 
time  depends  only  on  the  value  of  the  input  at  that  time),  differential  operators  (not  dif¬ 
ferential  equations,  but  such  direct  statements  as  "the  output  Is  the  derivative  of  the 
input"),  and  integral  operators  (among  which  are  the  integral  operators  of  Zadeh  (I  l|] 
are  also  specially  represented. 

An  implicit  functlor  method  (b),  is  an  equation  that  doe*  not  give  f(x)  directly  In 
terms  of  x,  but  specifies  s  condition  jointly  on  x  and  l(x)  so  that  for  any  x  there  is  a 
value  for  f(x)  thai  will  satisfy  the  condition.  A  differential  equation  ia  exactly  this  sort 
of  condition;  given  any  input  f,  it  is  nscesssry  to  go  through  a  process  called  "solving 
the  differential  equation"  in  order  to  obtain  the  output  g.  The  method*  de\ . aed  by  the 
Russian  schools  (or  obtaining  such  solutions  are  all  special  methods,  restricted  to  cer¬ 
tain  kinde  of  equations  and  certain  kinds  of  inputs,  just  a*  use  methods  of  solution  for 
implicit  teal  functions  are  all  special  methods. 

Approximation  method)  »c),  (d),  and  (e)  are  more  generally  applicable,  since  they 
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do  m  m«  ri  appeal  forma  tor  <U  rtyrtxaa'.m  «f  A*  hamow.  tiaM|k  tkqr  do 
require  too)  not  conditions  be  satisfied.  For  dM  ■rhuds  to  vUl  be  danairl,  a 
sunniest  nedtxa  tor  arbitrarily  don  approximation  is  dot  the  fvmctseo  (tot  ta  to  be 
repress  mad  be  cattunom  aod  have  a  comport  dooaur.  The  interpretation  of  three  coo- 
dittoes  for  sywtw n  nil  be  disrwswed  to  Section  Q.  The  mettnf  themselves  will  oca* 
be  ontSt  orsc  ribed. 

A  table  o<  values  (c)  is  ccoctned  of  here  as  being  used  to  the  simplest  possible  mac- 
ner.  that  u,  without  interpolation.  In  the  cocstrocuoo  of  the  table  a  finite  act  of  la 
selected  from  the  domain,  and  tor  each  aelected  xj  the  corresponding  ili^  i»  tabulated. 

In  the  aw  of  the  table,  tor  any  (nta  a  the  Dearest  tabulated  a(  u  aelerteo  and  the  cor¬ 
responding  fix)  1a  taxen  a*  ah  approximation  to  fix).  Owing  to  the  way  in  »K«c h  the  table 
is  used.  :ta  coarrucUott  can  be  modified.  Firat.  niece  each  tabulated  x  is  actually  used 
to  represent  a  set  of  neighboring  i't,  the  entry  m  tee  table  may  be  a  designs  turn  for  tlua 
set  instead  of  a  particular  x  is  the  set.  Second,  since  each  tabulated  flsj)  is  used  to 
approximate  a  set  of  fix),  the  tabulated  value  need  not  be  a  (urtiodir  fli^  but  may  be 
simply  a  ole  that  is  repreaerautrt  this  set  of  fix).  Either  of  these  schemes  can  be 
translated  uco  a  method  for  the  approximate  representation  of  functionals  by  replacing 
x  by  a  aad  f  by  h.  The  modified  scheme  is  then  a  general  description  of  Smgleton's 
method  for  approximating  nonlinear  syst  is  by  fume. state  transducers  110)  Bose's 
method  t i  representation  («)  also  employs  the  device  of  a  finite  table  of  nlaes.  Another 
method  inwol  vicy  tables  of  values  is  given  ui  Section  1C. 

An  abstract  definition  of  a  polynomial  Id)  t  ill  be  greet,  in  Section  IV.  as  hell  as  the 
particular  bra  of  polynomial  representation  that  aras  also  used  by  Wiener  (4). 
Wehara  IS),  and  Deotsch  (6).  For  our  presen'  purpose.  U  sufficient  to  note  that  the 
sum  of  a  constant,  a  linear  system,  and  products  of  linear  systems  (obtained  by  using 
the  same  input  for  all  systems  and  multiplying  -  -  adding  the  outputs)  is  a  polynomial 
system.  The  formula  used  ir.  Section  IV  is  lo  r*  at  more  general  than  Uus,  and  has 
been  found  to  be  convenient  for  the  computations  that  are  required  in  systems  analysis. 

Exruuiiicr-*  m  orthogonal  functions  (e)  will  be  discussed  in  Section  V.  These  methods 
give  promise  *»C  being  convenient  for  the  measurement  of  noclmexr  systems  tn  the  tano. 
ratory.  and  their  advantages  ran  be  combined  viti  the  computational  convenience  of  poly¬ 
nomials  by  using  expansions  m  ortbogixjJ  polynomials.  The  gene ral i xa uoc  of  these 
methods  from  real  functions  to  systems  is  quite  in’f restme.  As  we  lino*  from  the  theory 
of  real  functions,  expansion  ol  a  function  in  orthogonal  f me < tons  invol.es  integration  over 
the  domain  of  the  function,  latrgrauoc  over  •  >ei  of  real  numbers  is  a  fsouliar  process, 
but  ho*  can  *e  integrate  over  •  set  of  functions?  Definition  af  such  integrals  *as  the 
essential  problem  that  Wiener  (1)  attacked  m  l°U,  and  at  tost  time  it  wax  a  difficult 
problem.  No*,  however,  probability  theory  offers  a  solution;  on  a  statistical  ensemble 
of  functions,  which  is  just  a  se’  of  luacuons  with  probat>dit:e»  defined  or.  it.  an  ensemble 
average  iexpectauor.j  ls  equi^aim*  to  an  integral.  This  is  the  essential  reason  for  the 
introduction  of  probability  in  the  methods  of  Booion  (7)  and  Bose  IS),  as  well  as  in  the 
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method  of  Wiener  described  by  Booton  end  Bose;  these  can  be  !  .  ■  . 
of  expending  s  nonlinear  system  in  a  series  o.  orthogonal  system-. . 

The  following  sections  discuss  some  example:  if  approxlm- . 
cations,  and  some  sufficient  conditions  for  their  applicability  -ct 
ditions  of  spproximability,  and  the  neat  three  sections  are  dev.,.  .» 
methods  of  spp  roil  mat  inn. 
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11.  APPROXIMATIONS  TO  NONUTCEAft  SYSTEMS 


l. 1  TOPOLOGY  AND  APPROXIMATIONS 

The  crutary  aim  of  this  section  •_»  to  establish  toot  sufficient  conditions  for  the 
approximating  ot  *  nonlinear  system  by  the  method*  that  will  be  described  u  subsequent 
section*.  The  most  important  reulu  o t  tlu*  sec  tion  are  suru merited  in  section  £,  T. 

The  theorem*  that  will  be  developed  are  eoaenually  theorems  cl  analysts;  in  tact,  one 
theorem  of  analysts.  I1..  Stone -Wererrtrsss  theorem,  sill  be  quoted  and  used  wituout 
proof.  Mott  of  tl  mathtmaucal  ku  ran  be  found,  in  the  restricted  context  of  real 
funct'ons,  in  Ruuin's  ■Pnncinles  of  Matbeoatital  Analysis*  ill);  the  Stone. Wr.*r» trass 
theorem  that  he  proved  is  applicable  to  —■  purpose.  Tor  a  dtscussioa  of  sjalrsia  in  a 
more  general  setting,  especially  for  a  general  definition  of  a  topological  space,  and  for  a 
more  appropriate  definition  of  a  contact  set  than  la  given  in  Kudin,  referer-e  c.  ►# 
made  to  Hille'a  *Punctionl  Analysis  a  .d  Semi  -Groups*  (1)). 

One  way  In  which  a  topology  mar  be  rigorously  defined  is  it.  terms  of  oeighbu-Poods. 
A  topological  space  is  a  set  of  objects  x  in  which  certain  *uh'-*a  Nix)  an  destfoa'ed  as 
neighborhoods  of  specific  objects  x.  (Usually,  there  is  an  it-.. .it j  :  t  bjects  i  Ml,  for 
eich  x.  an  infinity  of  N(x).|  These  setghterbuuds  satisfy  certain  conditi.  is  that  rosli. 
tute  the  postulates  of  topology;  first,  every  x  has  at  lewst  one  Nix),  and  every  Nix)  con¬ 
tains  x;  second,  if  N^(«)  and  Kgix)  are  two  neighborhoods  of  the  same  object  x,  there 
i*  an  Nc(x)  wi  a  he  property  that  a=y  object  it.  Nc(x)  is  also  in  both  NAU)  and 
third,  for  any  cjj  ;ct  y  contained  m  set.  neighborhood  SA(x)  there  is  an  Npiy)  with  the 
property  that  any  object  in  Ng(jr)  aa  also  in  NA(x).  (Conventionally,  lhe  objev-.  in  a  topo¬ 
logical  apace  are  called  ‘points.*  Ttii  le~na  will  not  be  used  in  this  report  because  it 
suggests  a  very  restricted  interpretatxoa  of  topology.) 

It  wilt  now  be  shown  that  topology  aa  just  defined  is  a  mathematical  anal,  guc  of  the 
engineering  idea  of  approximation.  Practically,  approximations  occur  when  we  .-'wider 
some  object  (e.g. .  a  number,  a  position  in  space,  a  resistor,  a  signal,  a  system)  mat 
is  to  be  used  for  some  purpose,  and  want  to  know  what  other  object*  are  sufficiently 
similar  to  it  to  be  used  for  the  same  pot-pose.  We  thus  define  a  criterion  of  approxima¬ 
tion  to  this  object,  and  consider  tt>»  set  of  all  objects  that,  by  this  criterion,  are  good 
approximations  to  it.  It  will  be  shown  that  these  approximation  sets,  as  neighborhoods, 
sati.ify  the  postulate*  of  topology  mat  have  been  given. 

First,  every  object  considered  by  engineers  is  usable  for  some  purpose,  and  thus  at 
least  one  neighborhood  is  defined  for  it;  and  for  any  purpose  an  object  is  always  a  good 
approximation  to  itself.  Second,  if  aaooject  x  can  be  used  for  two  purposes  A  and  It, 
two  neighborhoods  N\(«l  and  Ng(x)  thus  being  defined,  we  coneider  purpose  C  as  the 
requirement  of  being  sufficiently  similar  to  s  to  satisfy  both  purpose*  A  tnd  B:  this 
defines  a  neighborhood  N^-fx)  with  the  property  that  every  object  In  hi^fx)  is  also  in  both 
hA<x)  and  Ng(x).  Third,  given  a  and  some  NA<x),  and  any  y  in  N  A<*l.  »e  con  consider 
purpose  T  for  j  ss  that  of  subsutwoag  for  s  in  the  fulfillment  of  purpose  A.  sad  esa 
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define  l<H(y)  as  the  ret  of  all  object*  that  are  sufficiently  similar  to  y  to  serve  tnl*  pur* 
pone;  then  every  object  in  Ng(y)  la  al*o  In  N^(x). 

These  argument*  may  seem  trivial  and  pointless;  actually  they  establish  the  relation 
between  topology  and  appro  i  mat  ions  and  make  the  topological  foundations  of  analysis, 
and  all  the  theorems  that  i allow  from  them,  applicable  to  engineering. 

For  any  aet  of  objects,  different  classes  of  approximation  criteria  can  often  be  uaed, 
with  the  result  that  different  acta  of  neighborhoods  and  different  topologies  are  obtained. 
However,  different  acts  of  neighborhoods  do  not  always  lead  to  different  topologlce.  Two 
■ets  of  neighborhoods  arc  said  to  be  bases  of  the  stmt  topology  If  every  neighborhood  in 
ich  set  contains  at  least  one  neighoorhood  from  the  other  set.  This  is  because  the 
closed  sets,  open  eets.  compact  sets,  and  continuous  functions  (defined  in  section  2.2) 
are  the  same  for  both  sets  of  neighborhoods. 

On  a  spue?  of  real  numbers,  a  neighborhood  of  a  number  x  is  defined  by  the  prop* 
erty  that  y  is  in  N((x)  if  the  magnitude  of  the  difference  between  x  and  y  is  less  than  (. 
In  the  uniform  topology  on  a  space  of  real  functions,  g  is  in  N{(f)  if,  for  every  real 
number  t,  the  magnitude  of  the  difference  between  f(t)  and  g(t)  is  lees  than  <;  a  similar 
condition  define*  the  uniform  topology  on  a  space  of  functionals.  On  a  apace  of  hyper* 
functions  (or,  equivalently,  systems),  we  define  the  uniform  topology  by  the  statement 
that  K  is  in  N  .(ll)  If.  for  every  Input  f,  at  every  time  t,  ti>e  magnitude  of  the  difference 
of  the  values  of  K(f)  and  H(f)  is  t  la  than  *;  or,  equivalently,  K(f)  is  in  N((ll(f))  for 
every  f.  A  different  topology  on  a  space  of  real  functions  will  be  defined  in  section  2.4. 

2.2  SOME  TOPOLOGICAL  CONCEPTS 

A  number  of  topological  ideas  that  are  to  be  uaed  in  the  discussion  of  approximations 
to  nonlinear  systems  will  now  be  defined.  We  begin  by  defining  open  and  closed  sets.  In 
spite  of  the  (act  that  wc  shall  make  no  use  of  them,  not  only  hecauae  mathematical  tradi* 
lion  seems  to  demand  it,  but  also  because  many  writers  define  topology  in  term*  of  open 
sets,  rather  than  in  terms  of  neighborhood*. 

An  open  set  is  s  set  with  the  property  that  every  object  in  the  set  ha*  at  least  ona 
neighborhood  that  is  contained  in  the  set.  A  closed  set  is  a  set  whose  complement  —  the 
set  of  all  objects  in  the  space  that  are  not  in  the  aet  -  is  open.  An  equivalent  definition 
is  that  a  closed  set  is  n  set  that  contains  all  its  limit  points.  When  a  topological  space 
la  defined  in  terms  of  open  sets,  neighborhoods  are  usually  defined  by  calling  every  open 
aet  a  neighborhood  of  every  object  that  It  contains, 

A  limit  point  of  a  set  A  is  an  object  (which  may  or  may  not  be  in  A)  every  neighbor¬ 
hood  >f  which  contains  at  lesat  one  object  in  A  other  than  x.  In  other  words,  a  limit 
point  of  A  is  an  object  that  can  be  approximated  arbitrarily  closely  (i.e.,  under  any 
criterion  of  approximation)  by  objects,  other  than  itself.  In  A. 

The  closure  of  a  set  is  the  aet  of  all  objects  that  are  either  in  the  set  or  are  limit 
points  of  the  act  (or  both).  In  other  words,  the  closure  of  a  set  A  is  the  set  of  ail  objects 


‘that  can  be  aRnrroxim.it  ed  arbitrarily  closely  by  objects  in  A.  In  the  application  of  this 
concept  *«  admit  consider  the  closure  of  ths  set  of  all  systems  that  can  be  exactly  repre. 
sented  by  sorme  .method,  the  closure  will  oe  tlie  aet  of  all  systems  that  can  be  represented 
>by  this  mathtnti,  cither  exactly  or  with  arbitrarily  small  error. 

A  mmpmr-  set  is  defined  as  follows.  A  collection  of  neighborhoods  ta  said  to  cover  a- 
:aet  A  if  evuury  object  In  A  is  in  at  least  one  of  the  neighborhoods  in  this  collection.  A 
-set  is  caileth  mumpact  if  every  collection  of  neighborhoods  that  covera  it  includes  a  finite 
sulicollrcuujn  tthat  also  covers  the  set.  If  we  define  a  criterion  Of  approximation  for  ovary 
‘Object  in  tint  *«t  A.  by  choosing  s  neighborhood  for  every  object  in  A.  this  collection  of 
'neighborhoiuua -covera  A;  and  if  A  it  compact  we  can  select  a  finite  set  uf  objecta  In  A 
with  the  pnjfT«stl;-  that  every  object  in  A  is  in  the  chosen  neighborhood  of  st  least  one  of 
the  selected) .objects.  The  importance  of  this  property  can  be  indicated  by  interpreting 
meighborhouittD  an  a  slightly  different  way;  that  is,  by  considering  a  neighborhood  of  an 
object  aa  *  uf  objecta  that  x  can  approximate,  Instead  of  -  aet  of  objecta  that  can 
.op,: /T-imatie  •*..  (These  interpretations  are  equivalent  if  the  approximation  criterion  has 
•th-  property  .Hint  x  approximates  y  whenever  y  approximates  x.)  Then  a  compact  set 
ta  one  u_tv,  iUir  any  predetermined  criterion  of  approximation,  can  be  approximated  by  a 
finite  aubsmttbf  itself. 

Topology  aa  combined  with  the  abstract  Idea  of  a  function  in  the  definition  of  a  con¬ 
tinuous  fumituon.  Suppose  the  range  and  domain  of  a  function  f  are  boih  topological 
spaces;  l  u*  said  to  be  continuous  if  for  every  x  in  the  domain,  and  for  any  neighborhood 
NA(f(x)l  uiTtihe  corresponding  f(x),  there  is  a  neighborhood  ND(xJ  with  the  property  that 
whenever  »  i»a  in  Ng(i|,  f(yi  is  in  N^lflxJ).  (Note  that  NA  and  Nq  are  neighborhoods  in 
'different  ajpaoes.i  This  is  a  pi  ecise  statement  of  the  imprecise  ides  that  a  continuous 
lunclion  u*>  unt  whose  value  does  not  change  abruptly;  it  implies  that  any  approximation 
' criterion. untthe  range  can  be  satisfu  oy  an  appropriate  criterion  of  approximation  in  the 
domain- 

2.J  TWa  HIUEOKEMS  OF  APPROXIMATION 

In  tarrmsuf  the  concepts  previously  defined,  two  important  theorems  on  approxima¬ 
tion  o<  fumumm*  can  be  stated.  These  theorems  wili  bs  applied  to  nonlinear  systems  in 
section  U..4i. 

The  firnc.  is  a  theorem  on  representation  by  tables  of  values.  Let  f  be  a  continuous 
function  mirth  a  compact  domain.  Let  a  neighborhood  N^fy)  be  chosen  for  every  y  in  the 
range.  Then  there  ia  a  finite  re*,  cf  objects  x]  in  the  domain,  and  for  each  r  s  neighbor¬ 
hood  Njjfw^J,  such  that  e»ery  x  in  the  domain  is  in  at  least  one  NQ(xt),  and,  whenever  x 
is  »n  f(xl  ia  in  N^fU^). 

To  uppity  this  theorem,  we  consider  functions  whose  ranges  are  sets  of  real  numbers 
or  real  fumctiuns,  such  as  functionals  or  hyperfunctions.  We  choose  a  positive  real 


number  <  a*  the  tolerance  for  a  cn'.enon  of  approximation,  and  define  neighborhood* 
in  the  range,  aa  in  section  2. 1.  Suppose  som-,  topology  1*  also  defined  in  the  domain, 
and  that  with  these  two  topologies  the  function  f  Is  continuous.  Then  we  can  select  a 
finite  aet  of  object*  x^  and  neighborhood*  Nl*iJ,  aa  indicated  ui  the  theorem;  we  construct 
a  table  of  these  x(  end  the  ■  irr*  a  ponding  fiXj).  Then,  for  any  x  in  the  domain,  we  can 
find  in  the  table  an  x^  wi  ne  property  that  x  ia  in  N(Xj),  and  the  tabulated  ffx^)  will 
differ  from  f(a)  by  less  .tan  «. 

The  proo*  of  this  theorem  is  quite  simple .  fines  the  function  la  continuous, 
there  is  a  neighbo-hood  Nglx)  for  every  x  with  the  property  that,  if  x*  is  in 
Nb(x).  f(x')  is  in  N^(f(x|).  The  collection  of  all  these  neighborhoods  N^ix)  covers 
the  domain,  and,  since  the  domain  is  compart,  there  is  a  Unite  set  of  x^  with  the 

property  that  the  collecUon  of  Nq(Xj)  al*i  covers  the  domain,  Thia  aet  of  x^ 

fulfills  the  conditions  stated  in  the  theorem,  and  the  theorem  is  thus  proved.  Incl* 
dentally,  we  have  also  proved  that  if  a  function  is  continuous  and  its  domain  Is 
compact  then  its  range  is  also  compact. 

The  second  theoiem  to  be  stated  here  ts  the  Stone. Weieratrass  theorem;  m  e'.Vc it 
t*  a  theorem  o  :  the  approximation  of  functions  by  polvnoouals.  I>  is  rostric.e.  .0  real, 
valued  fun'lim-e  Uthous.  th..  nature  of  the  domain  is  not  i“  "tr.f  icd.  U  if  at;  .  *r.  the 
fii  at  thee  cm  in  that  we  annum.  t_  .1  ’he  fu...'  t to  be  approximated  is  continuous  with 
compact  domain.  The  statement  of  this  theorem  must  o<  preceded  by  aome  preliminary 
definitions. 

If  i,  f|,  and  fj  are  functions  with  the  same  domain  and  A  la  a  real  number,  then 

f  •  f|  ♦  fj  if  fix)  *  f,|x)  ♦  f2<*)  for  every  x  In  the  domain,  f  »  f jf2  if  f(x)  ■  f((x)f2(x)  for 

every  x  in  the  domain,  and  f  »  Afj  if  f(x)  *  Af,(x)  for  every  x  in  Ihe  domain.  These 
definitions,  although  obvious,  arc  logically  nontrivial. 

An  algebra  of  function*  is  a  set  of  functions,  all  of  which  have  the  same  domain,  with 
the  property  that  for  every  f  and  g  in  the  set  and  for  every  real  number  A,  the  functions 
f*g,  fg,  and  Af  are  also  in  the  set. 

An  algebra  of  functions  is  said  to  separate  points  if.  for  every  pair  of  objects  x  and 
y  in  their  domain  and  »very  pair  of  real  numbers  A  and  D,  there  is  a  function  f  in  ths 
algebra  with  the  propeiiy  mat  fix)  ■  A  and  t(y)  »  B. 

The  Stone- Wcierstrass  theorem  states  that  if  an  algebra  of  real. valued  continuous 
functions  has  a  compact  domain  and  separates  points,  then  tne  closure  of  the  algebra,  in 
the  uniform  topology,  is  the  set  of  all  continuous  real-valued  functions  with  that  domain; 
i.e. ,  for  any  continuous  real. valued  function  f  with  that  domain,  and  any  positive  num. 
ber  (,  there  is  a  function  g  in  the  algebra  such  that  jf(x)  "  g(x)|  <  (  for  every  x  in  the 
domain. 

The  proof  of  thia  theorem  has  been  given  by  Rudln  (12);  it  Is  too  involved  to  be 
repeated  hen-.  Although  the  context  of  Rudin’s  proof  may  suggest  that  the  theorem  con* 
earns  only  functions  of  a  real  variable,  the  same  proof  is  vaild  for  compact  domains  in 
the  most  general  topological  spaces. 
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.4  A  SPECIAL  TOPOLOGICAL  SPACE 


The  approximation  theorem*  of  section  2.  S  will  now  be  siplied  to  nonlinear  systems. 
Specifically,  cihce  it  *ai  shown  in  Section  1  that  *  lime-invariant  system  ran  be  repre¬ 
sented  by  a  functional,  they  will  be  applied  to  functionals. 

The  theorems  indicate  that  *  sufficient  condition  for  «  function  to  be  approximate  is 
that  It  be  continuous  and  have  a  compact  domain.  These  properties  depend  upon  the  topol¬ 
ogies  on  the  domain  and  the  range.  On  the  range  (arhtch  is  s  set  of  real  numbers!  there 
is  only  one  useful  tonology;  but  on  the  domain  (which  is  a  set  of  real  functions!  a  fairly 
wide  choice  of  topologies  is  possible.  The  practical  meaning  of  the  theorems  depends 
upon  the  topology  that  is  used  on  the  domain;  but  if  the  theorems  are  to  have  any  practical 
meaning  at  all,  the  topology  that  is  used  must  imply  both  a  physically  meaningful  defini¬ 
tion  of  continuity  and  the  existence  of  physically  significant  compact  sets. 

A  topology  that  meets  these  requirements  has  been  found.  (Further  research  might 

reveal  others.!  On  the  space  P01(K),  which  was  defined  in  Section  1  as  the  eel  of  all  real 

functions  u  for  which  |u(tj|  a  R  for  all  t  In  the  domain  0  t  t  <  «e,  neighborhoods  &(u| 

are  defined  as  follows;  v  is  in  N._  ,(u),  T  >  0,  5  >  0,  if  and  only  if 

1,0 


v(t)]  dr  <  6 


(Z) 


for  all  x  in  the  interval  0  «  x  «  T.  The  topology  defined  by  these  neighborhoods  will  be 
called  the  RTI  (Recent  Time  Integral!  topology. 

This  condition  may  be  alternatively  expressed  by  defining  the  functions  U  and  V, 


Then  v  is  In  NT  &(u)  if  and  only  If  the  magnitude  of  the  difference  b  »en  U(x)  and  V(x| 
is  less  than  6  for  every  x  in  the  interval  0  <  x  <  T.  Note  thot  if  v  is  m  NT  j(ut,  then 
u  is  in  N^.  6(v>,  and  vice  versa.  It  will  be  seen  that  for  v  to  be  in  NT  j(u)  no  condition 
need  be  imposed  on  the  values  of  these  functions  for  t  >  T,  although  for  t  *  T  the  differ¬ 
ence  between  u(e)  and  v(r)  need  not  remain  small,  but  may  alternate  rapidly  between 
large  positive  and  negative  values. 

It  will  be  shown  in  the  next  section  that  the  space  I’BIIK),  for  any  R,  is  compact  in 
the  RTI  topology.  We  therefore  consider  a  functional  h  whose  domain  is  the  spa.-e 
FTI(R).  This  functional  is  continuous  if,  for  any  positive  number  «,  there  exisl  positive 
numbers  T  (sufficiently  Urge)  and  6  (sufficiently  small!  such  that  if  v  is  in  NT  g(u! 

(and  v  and  u  are  both  in  Pi)l(H)),  then  j{](u!  -  bU)|  <  c.  The  functional  will  then  be 
called  continuoua(R|,  and  'he  time-invarUnt  system  H  that  it  represents  will  also  be 
called  connnuous(R).  Any  system  referred  to  as  continuous  is  understood  to  be  time. 
Invariant. 


1 1 


b  the  representslior  at  systems  uy  fu,«.U.aiala.  the  function  a  rtprtMnti  the  past 
at  the  input.  We  may  th.-ief-re  Interpret  cot.unuity  tor  ncnluctr  system*  (with  respect 
to  the  RT1  topology)  by  the  statement  that  a  system  is  costintmu(K|  it.  for  lU  inputs 
that  are  bounded! R),  the  value  of  the  output  is  relatively  insensitive  to  small  changes  in 
tt.e  input,  to  rapid  fluctuations  (Mg  »  frequencies!  in  the  input,  and  to  the  remote  past  of 
the  input. 

it  follows  from  the  first  theorem  of  section  1. 1  that  a  system  H  that  is  continuogs(R) 
can  he  represented  with  any  desired  accuracy  by  s  Unite  table  of  values,  sir.ee  the  tunc, 
ticnal  that  represents  it  is  a  continuous  fi:"-uon  with  s  compact  domain.  Let  any  toler¬ 
ance  t  be  given;  then  T  and  %  are  determined  according  tc  the  continuity  condition,  a 
finite  set  of  real  functions  u^  is  selected  with  the  property  that  the  collection  of  neigh¬ 
borhoods  Nt  slut)  covers  P&NR),  and  these  real  functions  are  tabulated  with  the  cor- 
resprmaing  values 

It  wta  be  shown  in  section  2.S  that  a  Ume-umirisnt  linear  system  is  conUnuoue(R) 
tor  soy  R  if  and  only  if  its  impulse  response  is  Leo* ague  integraJole;  this  is  roughly 
equivalent  to  the  condition  that  its  transients  be  damped  and  that  its  impulse  response 
*1  voire  no  impulses.  The  set  of  all  such  linear  systems,  all  products  of  these  systems. 
«U  constant -output  systems,  and  all  sums  of  these,  ia  an  algebra.  The  functionals  that 
represent  them  constitute  an  algebra  of  continuous  .'unctionaU.  It  ia  easy  to  show  that 
this  algebra  separates  points .  The  Stone-Weierstnsa  theorem  then  Implies  that  any 
functional  that  ia  continuous  {R),  with  domain  PEU(R).  can  be  approximately  represented, 
with  arbitrarily  small  error,  by  a  functional  chosen  from  this  algebra.  Hence,  any  sys¬ 
tem  that  is  conUnucvs(R)  can  be  approximated  arbitrarily  closely  in  polynomial  form. 

Z.S  CONTINUITY  AND  COMPACTNESS  IN  THE  RECENT  TIME  INTERVAL  (RT1) 
TOPOLOGY 

This  secUon  is  devoted  to  proofs  of  two  statements  made  in  section  2.4;  that  a  time- 
in  variant  linear  system  is  conbnuous(R),  for  any  R,  if  and  only  if  Its  Impulse  response 
is  Lebesgue  tntegrsble;  and  that  tha  spscs  PBI(R)  Is  compact  in  the  RTI  topology. 

The  theorem  on  continuity  of  s  linear  system  will  be  proved  f.rat.  A  time -invariant 
linear  system  is  represented  by  the  functional  g,  defined  by 

W«)  .  /  h(»)  stir)  dr  (4) 

Jo 

where  h  is  ll:r  impulse  response  of  the  system.  Suppose  that  h  is  not  Lebesgue  inis- 
g  rafale;  this  may  be  so  either  because 


dr  •  m 


<« 
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h  U  not  abaoL  Integrabl*),  or  (kiuh  the  Integra)  of  h  la  an  defined  that  tt 
la  not  equal  to  the  Lebcsgue  integral  (e.g..  h  Involve*  inpulaca).  It  will  be  ahown  in 
each  of  theae  two  >  ma  that  R  t*  not  continuous!!*)  for  any  H. 

Suppose  fc  ia  not  ab*olutcl>  integrabl*.  Chooae  <  >  0,  and  try  to  find  a  T  and  I 
with  the  property  that  if  v  i*  in  h'T  &<u)  then  |t)(u)  -  h(v)|  <  *.  But  we  can  chooae  it  and 
v  ao  that  v(t)  -  u(t)  ha*  a  conatant  magnitude  lea*  than  6/T,  and  an  algebraic  algn  that 
la  always  equal  to  the  algn  of  !.(▼);  then  v  It  tn  N^.  4(u),  but  the  difference  between  h(u) 
and  },(  »-)  ia  infinite. 

Now  auppoae  that  h  contain*  an  impulse  of  value  A  (l.  e. ,  A  ia  the  integral  of  the 
Impulse)  at  *o>  Chooae  c  *#*a  than  |2AR|  and  try  to  find  a  corrciponding  T  and  I. 

But  If  wu  chooae  v  and  u  ao  that  their  value*  are  equal  except  on  a  email  interval  that 
contains  *0.  we  can  have  v  In  &(u)  by  making  tht*  Interval  small  enough  and  a  till 
have  1  u|t^)  -  v*to1)  ■  ZR,  with  the  result  that  |h(u)  -  fc(v)|  *  |ZAR|  >  c.  A  similar  argu¬ 
ment  hold i  whenever  the  impulse  response  is  absolutely  Integrabl*.  but  not  Lebeagu* 
integrablc,  since  in  that  cast  the  indefinite  Integral  of  the  impulse  response  la  not  abso. 
lutely  continuous. 

Now  auppoae  that  the  Impulse  response  h  is  Lcbesgue  integrabl*.  We  prove  that  h 
ia  a  continuous  functional.  We  consider  the  domain  of  the  functional  to  be  PB1(R)  for  any 
chosen  R,  and  choose  any  c  >  0.  Let  p  »  */«R.  Now  conetruct  a  step-function  hp  -  a 
real  function  whose  value  la  conatant  on  each  of  n  bounded  Intervals  and  ia  aero  outaid* 
them  -  so  that 


jT  |hp(t»-h<T)|  dr  «p 


<6) 


The  existence  of  such  a  step-function  can  be  proved  from  the  fundamental  definitions  of 
tne  Lebeagu*  Integral;  It  la  obvlou*  If  h  la  continuous.  There  ia  a  number  M  with  the 
property  that  |hp(v)|  k  M  for  all  t.  and  a  number  T  with  the  property  that  hp<»)  •  0  for 
all  v  >  T.  Let  6  •  a/t.iM. 

Let  v  be  In  Nt  &(u)  for  these  values  of  T  and  6.  Then  for  any  one  of  the  n  inter¬ 
val*,  say  a  <  ▼  «  b,  ws  have 


|  J  hp<T)  u(t)  d i  -  j  hp(v)  v(t)  d»J  s  J  j  hp(v)[u(T)  -  vfr)J  dr| 
<M  Lf  -  v|t  >1*|  t  ZM6  «  c/jlt 
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Slnca  th era  ar*  n  intervals, 

lbp(«»-yvi|  <*/j 


Also,  sine*  u  -a  bounded  and  hp  approximate*  h. 
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so  that  a*  we  |<u<  (ran  to  'jp(u)  to  h^tvl  to  bfv>  the  chinfti  ore  ail  mull,  and 

|H(U)  -  b(v)|  *  «/A  *  «/J  ♦  t/4  <  €  (10) 

and  the  continuity  of  ft  U  primed. 

We  turn  row  to  proving  the  statement  that  the  apace  PUi(fl)  is  cow  port  in  the  RTt 
topology.  The  proof  is  accomplished  by  redwctm  ad  absurdum ,  we  iwaat  that  the  space 
Is  not  compact  and  derive  two  contradictory  conclusion*. 

We  begin  by  constructing  a  special  set  of  functions  v  ,  In  the  «pr  t  PB1(R).  .  ne  set 
of  functions  vn  for  a  given  n  ts  constructed  by  -  idmg  tne  Interval  C  <  t  <  I"  into  l 
subintervals  of  length  2**;  the  value  of  each  linctinn  v  ,  is  constant  oa  each  subinterval 
and  equal  to  either  It  or  -R.  and  is  sero  far  v  >  2.  For  each  n.  the  index  k  will  there* 

ylZX 

fore  run  from  l  to  Z4  .  S*nce  the  number  of  these  functions  for  each  •  is  finite,  all 

these  functions,  for  all  a.  can  be  arranged  in  an  infinite  sequence  in  order  of  Increasing  n. 

With  each  v  we  associate  the  particular  neighborhood N*(v„  i.)*NT  ,{v_  .  ) defined 
ntK  |  _  *v»  n«k 

by  T  i  2  and  &  *  2*  -ft.  It  will  now  be  shown  that  for  any  n  tbf  collection  of  neigh¬ 
borhood*  N*(v  )  cover*  PBl(R).  Ttu#  is  equivalent  to  showing  that  for  any  u  in  PRI(R) 
n,  it 

snd  for  any  integer  n  we  can  construct  a  function  v_  .  in  such  •  way  that  the  magni- 
tude  of 

E(x)  mj*  (v^fv)  -  u(t)J  dr  (11) 

is  less  thar  21_nR  for  every  *  In  the  Interval  |(i<  2n.  We  conatrwrt  such  a  function 
»n  k.  starting  from  v  s  0,  by  the  following  procedure.  Suppose  that  v^  l'|  has  been 
determined  on  tbe  interval  Oitlr  .  We  shall  decide  whether  the  value  of  v  .  is  to 

°  -a  *'»* 

be  R  or  -R  on  the  interval  r  <  v  *  ♦  2  .  According  to  our  decision.  E(t)  will  either 

increase  or  decrease  mowotomcaUy  on  this  interval  to  a  value 


Since 


(»2) 


E(To>- 


I  C*!  <  l1_nR  ♦  i^R  *  J 


at  least  one  of  these  alternative*  will  make  tbe  magnitude  of  E  Zj  lee*  than  j'^R. 
Since  the  magnitude  of  ElTp)  is  also  less  than  1  “  n  R,  and  E  is  asoeotooic  on  this 


U 


interval,  the  same  bound  hold*  evei-ywhere  on  th*  utcml.  We  cm  continue  thla  con. 
struction  over  the  entire  length  of  the  Interval  0<t<  l",  and  prove  that  (or  any  n  the 
collection  of  N*(vr  cover*  PBltK). 

No*  suppose  that  (or  each  u  in  PBI(R)  we  specify  *  neighborhood  N'(u);  then  the 
collection  of  ail  N‘(uJ  cover*  PBI(R).  and,  since  »e  assui  »  that  FBl(R;  1*  not  compact, 
we  assume  that  these  N'fu)  have  oeer,  so  chosen  that  no  finite  subcoilection  of  theae 
neighborhood*  cover*  PBi(Rl.  We  can  prove  at  the  outset  that  there  is  a  countable  sub. 
collection  that  covers  PBi(R)  -  that  ts,  a  collection  that  can  be  ordered  in  a  sequence. 

For  each  u,  JCfuJ  <-  (u)  for  aome  T  and  t.  Choose  n  so  large  that  2n  >  T  and 

I  .*  »«®  %  * 

i  <  t/i.  There  Isiv,  .  w.th  the  property  that  N  (v„  .  )  contains  u,  and  this  N  (v  .  ) 
is  contained  in  N'(ul.  Thu«,  resigning  tome  N  (v  .  )  to  every  u,  we  obtain  a  aubcoliec. 

|  "i* 

tion  of  the  infinite  sequence  of  N  (vn  This  subcoUection  covers  PB1(R)  and  can  be 
arranged  in  a  sequence,  and  each  neighborhood  is  contained  in  at  least  one  NMo).  For 

each,  pick  one  of  the  N'(u)  that  contains  it.  We  thus  obtain  s  sequence  N’(um>,  m»  1,2 . 

which  covers  PB1(R). 

Now  choose  m(  a  l.  No  finite  collection  of  N'(u)  covers  PRliR),  so  there  is  some 

function  w.  in  PBl(it)  that  is  not  in  N'fu.),  out,  since  the  collection  of  NMu _>  covers 
ii  m 

PBitH),  there  Is  an  with  the  property  that  NMum2>  contains  w(.  Consider  neat  the 
collection  of  NMu^),  m  «  1,2,...,  m^.  This  collection  does  not  cover  Plll(R),  su  there 
is  s  w2  that  is  not  in  any  of  these  neighborhoods,  but  tnere  is  an  «n ^  with  the  property 
that  w2  is  in  N'iu,,,^).  Proceed,  in  this  manner,  to  construct  an  Infinite  sequence  of  w(, 
with  the  property  that  no  finite  collection  of  the  neighborhoods  NM>»m)  contains  more  than 
a  finite  number  of  them. 

We  shall  now  contradict  this  conclusion  by  proving  that  for  any  sequence  of  functions 
in  PBi(R)  there  is  at  least  one  neighborhood  N'fum)  that  contains  an  infjfiite  number  of 
Wj.  Consider  uie  collection  of  N*(vn  for  n  «  2.  This  collection  covers  PB1(R)  ar.d 
contains  a  finite  numbe-  of  neighborhoods,  so  that  *t  least  one  of  them,  which  we  ahall 
call  N  (v  ),  contains  an  infinity  of  w..  If  we  consider  only  the  w  that  arc  contained  In 

g  "  f  *  *  g 

N  (v2),  the  collection  of  N  (vft  ^ )  for  n  »  3  must  contain  at  lcaai  one  N  (Vj)  that  contains 

an  infinity  of  these  w..  We  proceed.  In  this  way,  to  construct  a  sequence  of  neighbor. 

0  * 

hood*  N  fvj),  each  of  which  contain*  an  infinity  of  Wj,  including  any  w^  lhal  i*  contained 
in  any  N  (v^)  that  follows  It  In  the  sequence. 

We  now  define  the  function*  Vy 


it  can  be  shown  that  for  every  a  the  sequence  of  numbers  V^xt  form*  s  convergent 

•equenee,  so  these  functions  V.  have  a  limit  function  V  .  it  car.  also  bt  shown  that  V 

jo  o 

is  absolutely  continuous,  and  that  the  magnitude  of  its  derivative  does  not  exceed  R,  so 

there  is  a  funcM-m  v^  in  PBJ(R)  with  the  property  that 


IS 


ns) 


v«i  •  jf*  vr>  * 

It  viU  to  seen  that  every  neighborhood  at  t#  mmm  m  tkfiut}  of  w(.  Store  there  to 
•1  least  on*  N'(um)  Uut  contain*  *o,  and  tttij  — ughboebood  that  contain*  »o  contain*  a 
neighborhood  of  vQ.  the  NMu^)  contain*  an  mfmity  of  w^.  Thi*  statement  contradict* 
our  prenoui  conclusion.  The  assumption  that  PBVR)  ia  not  compact  bn*  led  to  a  con¬ 
tradiction.  and  we  conclude  tha'  PBMR)  to  conpwt. 

Z.  *  HYSTERESIS 

Hysteresis  i*  often  mentioned  a*  a  typical  phw  —non  of  nonlinear  ay* tenia.  Exam¬ 
ination  of  method*  of  approximate  representation  of  nonlinear  ey  a  terns  leads  to  a  general 
impression  that  these  methods  tail  whan  byateeeat*  occurs.  Hem’ ever,  a  general  defini¬ 
tion  of  hysteresis  ia  necessary  before  this  impress***  can  be  promoted  to  the  status  of  a 
conclusion.  Such  a  definition  will  be  proposed  m  this  section,  it  will  then  be  a  ho  am  that 
syuicms  that  exhibit  the  phenomenon  of  hysteresis  are  not  contmuour.  in  the  sense  in 
which  continuity  was  defined  in  section  2.4. 

Tbe  phenomenon  of  hysteresis  to  usually  wderwaood  in  terms  of  a  hysteresis  loop. 


r  > 
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rig.  1.  Hysteresis  loops,  ft)  ho  hysteresis,  (b)  With  hysteresis. 

<c)  Two  systems  with  byatertsus.  (d)  Sum  of  the  two  sys¬ 
tems  in  (c). 
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A  system  that  adhibit*  hysteresis  I*  contrasted  with  a  system  in  which  th*  instantaneous 
value  of  th*  output,  y,  1*  uniquely  determined  by  the  inatantaneou*  value  of  the  input,  a, 
a*  in  Fig.  la.  Tht*  t*  a  typical  no-memory  ayatem. 

A  ayatem  with  hvatercala  la  then  conaidered  aa  on*  that,  for  a  given  vatu*  or  th* 
input,  may  have  on*  of  several  different  output  value*.  The  ayatem  ia,  at  any  time.  In 
one  of  aeveral  statr*.  the  atate  depending  not  only  on  the  value  of  the  input,  but  al*o  on 
previous  atatea  of  the  ayatem;  th*  atate  depends,  however,  only  upon  th*  order,  and  not 
upon  the  time,  in  which  these  previous  elates  were  passed  through.  /,  graph  repre¬ 
senting  such  a  ayatem  la  shown  In  Fig.  lb.  (For  the  sake  of  simplicity,  the  aides  of  th* 
loop  are  made  vertical.)  When  a  changes,  y,  if  the  graph  indicates  nmre  than  on*  pos¬ 
sible  value,  assumes  the  value  nearest  to  the  value  it  had  most  recently.  The  vertical 
parts  of  tne  loop  can  therefore  be  traversed  only  In  the  Indicated  direction. 

In  the  ayatem  of  Fig.  lb  the  slate  of  the  ayatem  can  be  determined  from  th*  Input 
and  output  values,  but  this  la  not  always  true.  Consider  the  ayatem  of  Fig.  Id,  which 
1*  formed  from  the  two  systems  of  Fig.  1c  by  applying  the  asm*  Input  to  each,  and  adding 
the  outputs.  If  the  two  loops  of  Fig.  lc  were  of  the  same  height,  points  A  and  A*  of 
Fig.  Id  would  Indicate  the  same  input  and  the  same  output,  but  would  represent  different 
etates. 

In  such  simple  caees  as  these,  we  can  say  that  the  output  of  a  hysteretic  ayatem 
depends  on  its  past  history,  whereas  the  output  of  a  nonhyateret  c  ayatem  does  lot.  But 
thir  statement  la  not  sufficient  In  a  general  context,  when  f  tput  of  a  nonh/Sierelic 
ay  cem  (e.g.,  a  linear  ay  at  ~m)  also  depends  on  the  past  of  the  input.  To  define  hyster¬ 
esis  for  general  time. invariant  systems,  we  use  two  conditions  that  may  be  deduced 
from  the  graphs  of  Figs,  lb  and  id,  but  are  meaningful  also  in  a  general  context. 

In  a  hysteretic  ayatem  It  is  possible,  first,  to  specify  two  inputs,  different  for  t  <  0 
but  equal  for  t  >  0,  for  which  the  difference  between  the  corresponding  outputs  does  not 
converge  to  cero  a*  t  —  «.  Second,  it  ts  possible  to  Specify  sn  input  for  t  >  0  In  *uch  a 
way  that  for  any  two  inputs,  arbitrarily  different  for  t  <  o  but  with  the  specified  form  for 
l  >  a,  the  difference  between  the  corresponding  outputs  always  converge*  to  rero  a* 
t  —  «.  (The  condition  of  convergence  toward  zero  la  used,  instead  of  equality  to  zero, 
to  allow  for  dependence  of  the  output  on  the  finitely  remote  past  of  the  Input.) 

Any  system  that  ta  conlinuous(R)  ia  an  example  of  a  system  that  does  not  satisfy  the 
first  condition,  since,  for  any  «,  we  specify  T  as  in  the  definition  in  section  2.4,  and 
the  outputs  for  t  >  T  roust  differ  by  less  liian  (.  An  example  of  a  ayatem  that  satisfies 
the  rirct  condition,  but  not  the  second,  is  an  ideal  linear  integrator. 

We  can  prove,  by  using  the  principle  of  superposition,  that  a  linear  system  must  be 
nonhystereUc.  If  two  inputs  are  different  for  t  <  0  and  both  are  zero  for  t  >  0,  then 
adding  to  each  input  a  specified  component  for  t  >  0  can  only  add  to  each  output  a  com¬ 
ponent  that  is  the  same  for  both  outputs,  and  cannot  change  the  difference  between  the 
outputs.  Hence  if  a  linear  system  satisfies  the  first  condition  for  hystarssis,  it  cannot 
satisfy  the  second. 
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A  system  capable  of  aubharmonlc  reaponae  to  a  sinusoidal  lnpen  laUilitl  the  first 
condition  of  hysteresis,  atnce  two  Inputs  that  are  equal  (or  t  >  0  may  ptt  rise  to  aub- 
harmonic  outputs  in  different  phase.  If,  in  such  a  system,  the  output  tends  to  aero  when 
the  input  becomes  and  remains  tero,  ttie  second  condition  is  satisfied  and  the  system  Is 
hyttereiic. 

Since  a  hysteretlc  system  is  not  continuous,  it  cannot  be  i^wuwxd  arbitrarily 
closely,  in  the  uniform  topology,  by  continuoua  system*.  However,  (or  some  hyateretic 
systems  and  some  input  ensembles,  it  may  be  possible  to  find  cominwoua  system#  that 
approximate  the  output  with  arbitrarily  small  error  except  (or  an  arbitrarily  small  prob¬ 
ability  of  large  error.  The  reason  is  that  the  aeco  id  condition  in  the  definition  of  hys¬ 
teresis  Implies  that  events  can  occur  in  the  input  that  make  the  system  forget  what 
happened  before  them.  For  some  ensembles  and  inputs  there  can  be  a  positive  probability 
of  such  an  event  occurring  in  a  bounded  time  interval,  and  this  probability  will  approach 
unity  ns  the  length  of  the  interval  increases;  thus  s  system  whose  output  depends  effec¬ 
tively  on  a  finite  portion  of  the  past  of  the  input  may  be  made  to  approximate  the  output 
with  arbitrarily  small  probability  of  large  errur. 

2.7  SUMMARY 

The  principal  ideas  and  conclusions  are  outlined  in  this  section.  These  ideas  form 
i  the  foundation  of  the  methods  of  representation  that  wilt  be  discussed  ia  subsequent 

sections. 

The  input  to  a  system  is  represented  by  a  ri-1  f’-iction  f;  the  value  of  the  Input  at 
time  l  is  f(t).  The  output  is  similarly  represented  by  a  real  function  g.  If  f  la  Lebeague 
measurable,  and  |f,t)|  «  R  for  all  t,  then  f  is  said  to  be  boundcdfR). 

A  nonlinear  eystem  H  la  a  function.  It  assigns  to  every  input  f  (in  a  specified  set) 
a  corresponding  output  g  »  H(f). 

A  time-invariant  system  1]  can  be  represented  by  s  functional  h.  The  value  of  the 
output  at  time  t  is  determined  from  the  function  ut,  defined  by  u^t)  *  f(t-r),  t  >  0.  then 
h(u()  *  g(t).  If  f  ia  boundediR),  then  u{  is  an  element  of  the  set  of  functions  called  PBI(R).  4 

A  topology  is  a  scheme  of  approximation  criteria;  a  neighborhood  is  a  set  of  approxi¬ 
mations  satisfying  some  criterion.  On  the  space  PBl(R)  we  define  the  RT1  topology  by 
the  condition  that,  for  T  >  0,  6  >  0,  the  function  v  is  in  the  neighborhood  N^.  j(u)  of  the 
function  u  if  and  only  if 

|  f*  [u(T>  -  V<T)]  d»|  <  ft  (16) 

whenever  0«»<T, 

A  time. invariant  system  K,  considered  only  for  inputs  that  are  bounded(R),  is  said 
*  ■<  to  be  conlinuous(K)  if,  for  any  «  >  0,  thers  exist  T  >  0,  6  >  0  (T  Sufficiently  Urge,  t 

sufficiently  small)  such  that  if  u  and  y  art  in  PBI(R)  and  *  is  is  N^.  g(u),  then 
|h(u)  -  h(v)|  <  t. 
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It  U  la  contlnuoug(R),  then  for  any  c  >  0  there  is  a  finite  aet  of  function*  in  rilt(R). 
and  a  neiRhliorhood  N^.  &(Uj)  of  each  u^,  such  that  every  u  in  PBI(R)  is  in  at  least  one  of 
these  neighborhood*,  and  if  u  is  in  N-j.  ihen  |h(u)  ~1j(Uj)|  <  c. 

If  la  conUnuous(K),  then  for  any  (  >  0  there  ia  a  polynomial  ayatem  |J  ,  consisting 
of  a  sum  of  a  constant,  a  linear  system  with  Lebesgue  integrable  impulse  response,  and 
products  of  such  linear  .systems,  such  that,'  for  any  input  that  is  bounded(R),  the  values 
of  the  outputs  of  11  and  never  differ  by  more  than  (. 
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Itt.  A  DIGIT  Al.  APPARATUS 


'  Tit  RORY 

Aj  en  illustration  of  lomi  of  the  results  of  Section  U.  a  description  and  discussion 
of  a  hypothetical  apparatus  for  the  analysis  and  synthesis  of  nonlinear  systems  will  be 
presented.  The  apparatus  U  designed  for  the  laboratory  examination  of  an  unknown  ays. 
tem  and  the  synthesis  of  an  approximately  equivalent  system.  It  has  not  been  built 
because  it  does  not  appear  to  be  practical,  but  some  of  its  principles  are  interesting. 

The  apparatus  is  based  on  the  approximate  representation  of  functionals  by  means 
of  tablet,  of  values.  To  represent  a  functional  h,  we  tabulate  the  valuea  of  hlu)  for  a 
finite  number  of  real  functions  u.  The  functions  for  which  values  are  tabulated  in  thla 
appiratus  are  similar  to  the  functions  vR  ^  used  in  the  proof  of  a  theorem  ir.  Section  U. 
They  are  cur.iitructed  by  quantizing  time  m  interval*  of  equal  length  q  and  by  maxing  the 
value  of  the  function  constant,  and  equal  to  either  R  or  -R.  op  each  interval.  We  shall 
call  such  functions  quantized  functions.  They  can  be  represented  as  sequences  of  binary 
symbols,  1  for  value  R  and  0  for  -R. 

Given  any  input  function  f  that  is  bounded(P),  we  generate  a  quantized  function  f*  to 
approximate  It  In  the  sense  that  [f(t)  -  f  »(t)J  dtj  is  made  as  small  as  possible  for 

every  A  and  B.  The  value  of  f*  on  each  q-interval  must  obviously  be  determined 
later  than  the  beginning  of  that  interval,  without  any  knowledge  of  the  values  f  will  nave 
on  that  interval.  We  may  use  a  feedback  method,  ■■'nd  the  error  ol  approximation  can  be 
determined  by  means  of  an  integrator.  The  input  to  the  integrator  will  be  f  -  f*,  and 
the  output  will  be  the  error  signal  e. 


eft)  »  f  (f(x>  *  f  *<x)|  dx 


<W) 


and,  at  the  beginning  of  each  q. interval,  we  make  the  value  of  f  •  equal  to  R  on  that 
interval  if  c(t)  is  positive,  and  equal  to  -It  if  e(t)  is  negative;  if  eft)  »  0  we  make  an  arbi¬ 
trary  choice. 

tin  an  interval  of  length  q,  f  •  will  contribute  a  change  of  magnitude  qR  to  the  value 
of  c,  with  sign  opposite  to  the  sign  of  e  at  the  beginning  of  the  interval,  whereas  f  will 
contribute  an  unpredictable  change  of  mangitude  that  does  not  exceed  qR,  so  that  the 
vjlue  of  e(l)  will  be  kept  within  the  bounds  cf  ±2qR.  Then 


|/B  (Ht)  -  f*(t||  dl  *  * c f A>  ••  e(B>|  <  4qR 


<l«) 


This  error  is  attributable  partly  to  quantization  and  partly  to  the  delay  in  using  the 
values  of  f  to  determine  the  values  of  f*.  In  deriving  from  f*  in  approxin  ation  to  u. 
it  is  therefore  oestrablc  to  make  use  of  the  fact  that  (•  can  be  predicted  up  to  the  eno  of 
the  present  interval.  We  define  u*(t)  •  f  *(t*  -  t),  where  l"  is  the  end  ..f  the  q-interval 
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can  taming  t.  This  m«wt  tel  ij(*)  ia  rautam  an  interval*  of  ltti|th  %  M»mt|  from 
v  •  0,  sad  a*  nlMi  of  u*  on  tkm  taurnl*  art  tbt  nlu*  already  determined  for  f •, 
btpmta|  with  the  motl  I  tcttU  Then  for  nil  t  and  all  a. 


If  |«t<T>-uJ<T))d»|  <4qK 


Given  a  system  ]J  that  U  castmuoua(R).  and  given  c  aa  the  desired  tolerance  for  the 
output,  determine  T  and  4  to  aaUafy  the  continuity  definition,  and  lei  4  a  4/4R.  Let  a 
be  the  smallest  integer  that  ia  hot  lcaa  than  T/q  (t.e. ,  the  number  of  internals  of  length 
q  needed  10  cover  an  interval  of  length  T),  u*  need  J*  defined  only  on  the  first  n  inter* 
vale  of  length  q,  and  may  be  talcs  aa  aero  fer  v  >  nq.  Then  u*  ia  in  N^.  4fu().  and 
therefore  (£<u*)  -  l)luf||  «  c. 

The  function  u*  can  be  represented  aa  a  a  queace  of  n  binary  digits.  It  must  be  one 
of  2°  possible  function*  u*.  each  of  which  ia  represented  by  a  binary  number.  For  each 
number  we  tabulate  |j(u*J. 

In  many  cases  it  is  not  practical  to  determine  in  advance  the  appropriate  T  and  4 
for  a  given  (.  '*  such  cases,  a  and  q  can  be  chosen  arbitrarily,  and  il  the  resulting 
error  turns  out  oe  too  Large,  it  can  always  be  made  smaller  by  choosing  q  smaller 
and  oq  larger. 

1.2  CONSTRUCTION  AND  OPERATION 

Two  devices  are  conceived  for  the  implementation  of  this  theory,  aa  analysis  device 
(for  as  mi  rung  a  system  in  the  laboratory)  and  a  synthesis  device.  The  synthesis  device 
would  consist  of  two  parts,  one  a  quantiser  to  determine  u*.  and  the  other  a  storage 
device  that  contains  previously  determined  values  of  h^u*)  and  produces,  at  every  time 
t.  the  value  £’  ,,  The  analysis  device  generates  quantised  inputs  f,  so  that,  at  certain 

times  t.  ut  his  the  form  of  u*.  and  records  the  values  of  the  outpu  at  these  times  as 

fclu*). 

Fpure  2  illustra  e«  the  quantiser.  The  interval  length  q  is  determined  by  the  fre- 
quei.i  of  ;  e  pulse  generator,  sod  n  is  the  number  of  stages  in  the  aloft  register.  The 
<hilt  :« *;vr  holds,  at  ail  umea,  the  last  n  digits  of  (*,  and  deli.-ers  these,  as  u*,  to 
tb*  a.*.,  r  unit,  which,  in  turn,  delivers  the  output.  The  curve*  in  Fig.  2  were  calcu¬ 
lated  ..  „  .cate  typical  operation  of  tier  quantiser. 

The  an  ysia  device  consists  of  two  parts,  a  generator  that  generates  quantised  inputs 
and  a  suras*  unit  to  record  to*  output  values.  In  using  Uus  device.  It  is  not  oecessary 
to  wait  for  the  system  that  ia  being  tested  to  return  to  its  rest  state  before  every 
sequence  cl  a  dig  ’-s  of  input  because  ibe  system  is  sensitive,  within  the  specified  toler¬ 
ance,  only  to  the  last  a  digit*  of  any  long  input  sequence.  We  can  therefore  use  a  long 
ai'quetace  of  digits  with  the  property  mat  everv  possible  sequence  of  n  digit*  is  contained 
mu.  r  example,  the  sequetce  Ot  100  contain*  the  two-digit  groups  01,  M,  U,  00  in 
that  order;  similarly  the  sequence  014001 1101  contains  all  possible  seq-jeaeea  of  three 
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Fig.  2.  Quantizer. 


digits.  Alternatively,  we  can  use  a  random  sequence  of  digits;  the  probability  is  unity 
that  a  random  sequence  will  eventually  include  all  possible  sequences  of  length  n.  A 
random  sequence  might  be  generated  by  using  the  quantizer  with  a  shite -noise  input. 

The  use  of  random  inputs  «■  ggests  the  application  of  the  apparatus  to  the  synthesis 
of  fitters  according  to  statistical  criteria  <e.  g. ,  tor  an  ensemble  of  iniiuts  consisting  of 
a  signal  corrupted  by  noise,  in  order  io  minimize  the  mean-square  difference  between 
the  signal  and  die  output).  The  functions  (which  represent  the  past  of  the  input)  will, 
in  general,  form  a  probability  ensemble,  and  for  each  u(  the  value  dll)  of  the  desired 
output,  uf  which  the  output  gfi)  of  the  system  is  to  be  sn  estimate,  will  have  a  conditional 
probability  distribution.  Ttv-  optimum  estimate  can  be  derived  by  ore  of  many  possible 
criteria  from  this  conditional  distribution.  In  many  cases  we  can  approximate  the  dis¬ 
tribution  of  d(t)  conditioned  on  u(  by  the  distribution  of  d(l)  conditioned  on  the  quantized 
approximation,  u*,  to  u  .  We  m.ght  therefore  use  a  long  sample  of  tht  input  and  the 
dcs.red  output,  put  the  input  into  the  quantizer,  and  for  each  u*  record  samples  of  the 
corresponding  values  of  the  desired  oi  tiui.  If  the  sample  Is  large  enough,  we  obtain 
many  samples  of  the  desired  output  (or  each  u*.  We  estimate  the  conditional  distribution 
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Inm  these  —  ftH.  compote  me  optimum  estimate  of  the  desired  output  from  thto  die. 
(nbuuac,  *=i  iwt:f  the  optvtm  Hnmw  so  fc(u*).  For  tt.'_t  eoumetvoo  crtterU  there 
star  caster  vers  to  Server  the  optimum  ctunuu  from  the  samples;  the  process 
described  is  ptertl* 

3. 3  EXAMPLE 

We  m-m  cikviitr,  tor  a  eery  staple  system,  the  requirements  on  a  and  q  for  syn¬ 
thesis  of  tee  system  by  mesas  of  the  apparatus  described  in  the  preceding  section.  Con. 
aider  the  isetr  system  with  trepexy  responst  t/(sa),  that  is.  with  impulse  response 
«  .  This  ts  a  hopiM  filter  with  unity  low -frequency  gain. 

For  this  system. 


y  i)  *  f*  ae'  *  ”  j(t) 


The  system  is  coctLnuouslR)  (or  every  R.  Suppose  v  is  in  N^.  ^(u).  Then  whenever 
0  <  i  <  T.  we  hare 


j/  J»|r»  -  rfe>l  drj  <  i 


*  f**  I 

•►'aj  -ht«V  «  ta  J  —  rfr»J  drj 

*  e“aT(v|t»  drj  ♦  a  jf"  w-*"*"! J ule) |  ♦  jvfT)|Jdr 

4  s!le"*T(UtT)  -  V(r))£  ♦  a*  jfT  e^Utr)  -  V(r)J  dr|  ♦  ZR  e~*T 

ctltZR  e~*T  <Z2) 

in  which  we  hare  integrated  by  parts,  using 

Cf»)  -jf*  s(t)  dr,  VUI  -  j£*  vfvj  dr  <i3] 

Next,  haring  selected  t  as  the  tolerance  on  the  output,  we  choose  T  end  I  » j  that 

tiiU  e~*T  «  g  (24) 

and  act  q  a  1/tK,  a  «  T/q.  ao  that 


4qa  ♦  2  e"****  r  c/R  (25) 

It  ta  difficult  to  minimise  the  storage  that  is  required  (i.e.,  to  minim  lie  n),  but 
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kjrae  bound*  sre  easy  to  obtain.  T  mult  t>«  greater  than  (l/a)  log  (ZR/O.  tod  q  must 
be  lea*  than  </4aR;  hence,  n  ■  T/q  mutt  be  greater  than  (4R/«)  log  (ZR/«|,  which  gives 
a  lower  bound  on  a  (log  *  natural  logarithm).  We  obtain  an  upper  bound  on  the  minimum 
n  b>  finding  the  a  that  la  required  if  we  arbitrarily  chooae  T  and  (  no  that  at  ■  IRe"4^. 
We  obtain  T  •  (I/a)  log  (4R /«).  q  ■  c/taR,  and  a  ■  (SR/<)  log  (4R/<). 

U  we  act  a  tolerance  of  10  per  cent  (R/<  •  10),  then  1Z0  <  n  <  300.  approximately, 
and  the  minimum  number  of  stored  values  required  le  between  10J6  and  lO**^. 


3.4  CONCLUSIONS 

For  R/«  ■  10,  a  act  of  11  numbers  can  be  so  determined  that  the  output  will  always 
be  within  «  of  at  least  one  of  them.  The  requirement  of  10 stored  values  seems, 
therefore,  to  Imply  a  very  Inefficient  echeme  of  ayntheaia.  Part  of  this  large  storage 
requirement  arises  from  the  fact  that  many  widely  different  functions  u  will  have  the 
same  h(u),  and  this  grouping,  since  It  depends  on  the  system,  cannot  be  built  Into  the 
apparatus  m  advance.  The  same  value  of  |)tu)  therefore  must  be  stored  separately  for 
many  different  functions  u.  However,  there  are  still  several  sources  of  Inefficiency 
that  might  be  corrected. 

The  inefficiency  of  the  apparatus  arises  partly  from  the  fact  that  u*  la  derived  from 
a  quantised  approximation  to  f.  This  Imposaa  certain  constraints  on  the  quantised 
approximation  to  u(.  It  can  be  ehown  that  If  the  quantized  approximation  u*  la  derived 
directly  from  ut>  so  that  these  constraints  are  not  imposed,  we  can  construct  for  every 
u  in  PBi(.t)  a  u*  with  the  property  that 


J  J  lu(r)-u*(v)]  dr  <qR 


126) 


Instead  of  4qR.  Such  a  u*  can  be  constructed  by  using  n  linear  filters,  of  which 
lh  * 

the  a  filter,  k  ■  1,Z . n,  has  an  Impulse  response  that  la  1  from  aero  to  kn 

and  sero  thereafter,  and  by  quantising  the  output  of  each  fUtar.  Then  q  can  bs 

chosen  to  be  tour  times  as  large  for  the  same  T  and  6,  and,  in  the  example  of 

9  23 

section  1.3,  the  minimum  number  of  stored  valuee  will  be  between  10  and  10  , 

(Although  the  quantised  output  of  the  filter  has  k  +  I  possible  values,  and  hence 
?n(n+!)l  conceivable  combinations  of  values,  only  Zn  of  thest  combinations  are 
possible  tf  the  input  le  bounded(R),] 

There  la  another  source  of  Inefficiency  that  might  be  removed.  Although  data  on  the 
past  of  the  input  have  been  taken  only  from  a  bounded  interval  of  the  recent  peet,  as 
much  data  have  been  taken  from  the  more  remote  parte  of  this  interval  as  from  tne  more 
recent  parts,  and  this  .a  unnecessary.  We  can  correct  this  by  making  the  q-lntervals 
longer  (or  the  remote  past  than  for  the  recent  past.  This  can  be  done  almply  by  omitting 
some  of  the  filters  from  the  eet  described.  This  would  effect  a  further  substantial 
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reduction  in  the  camber  el  shared  values  that  is  required. 

The  apparatus  that  is  thus  evolved,  ceiuiftiAf  of  linear  filters  with  quantised 
outputs,  is  practically  simitar  to,  although  different  tn  derivation  from,  the  appa* 
ratus  proposed  by  Bose  ftp 


IV.  ANALYTIC  SYSTEMS 


»■ 

i 


4.1  THEORY 

On*  pi  the  conclusion*  of  Section  U  was  that  any  system  thst  is  continuous(R)  csn  b* 
approximated  arbitrarily  closely  by  a  polynomial  system  constructed  from  llnssr  ays. 
ttms  whose  Impulse  responses  sre  Lebesgu*  lntegrabls.  This  polynomial  system  is  a 
sum  of  a  constant,  a  linear  system,  and  products  of  linear  systems. 

The  output  j»  constant  system  can  be  represented  by  a  fixed  real  number;  the  out. 
put  of  a  linear  system  can  be  represented  by  means  of  a  convolution  integral.  For  a 
product  of  linear  systems,  whose  Impulse  response  functions  are  k,,k, . K,  we  have 

14  n 

KO-r...  r  ft  kl<Tl>  ft  m-  T.Jdt  ,-..dr  (27) 

J-m  J-*  let  1  1  1st  1  1  " 


where  f  represents  the  Input  and  g  the  output.  Or,  if  we  writs 

h„<T . .  s  kjtfj)  k2(T2)  ...  kn(Tft) 

we  have 


f(t  -  Tj)  dvj...dvn 


(24) 


(29) 


This  Is,  at  least  in  a  restricted  sense,  a  system  of  n^  degree,  sines  multiplying 
the  Input  by  s  constant  A  results  In  multiplying  the  output  by  An.  The  function  hQ  (which 
is  Lebesgue  lntcgrable)  will  be  called  the  system  function. 

The  eum  of  two  n01  degree  systems  Is  an  nth  degree  system  whose  system  function 
is  the  sum  of  the  system  functions  of  the  summands.  Therefore,  the  representation  of 
s  polynomial  ayatem  requires  only  one  term  of  each  degree: 


g(t)  •  hn  ♦  J  h,(T)  fC-r)  dv  ♦  a:  hZ*Tr  Tl*  l*‘  ~  Tl*  ”  t2*  <*tI<,t2 

*•••  . v  n, f<t - v dTr- dTi» 


Any  continuous  system  csn  be  Spproxlmsted  arbitrarily  closely  in  this  form. 

The  domah  ..  'cated  in  these  integrals  has  been  chosen  as  {-w>, »)  so  thst  physically 
unreallcablt  systems  csn  also  be  represented.  For  s  physically  reslissbls  system,  each 
system  function  hR  will  have  the  value  tero  whenever  any  of  its  arguments  la  negative, 
and  the  Integrals  can  then  be  taken  on  (0,w). 

For  convenience  in  certain  computations,  ws  define  the  system  transforms  Hn  by 

,ln‘*l . V'l.-LV’l . V”*'  *ri'-'Vfl,dT|-d,o 
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(In  particular,  Hq  a  hg).  Since  the  lyittn  function  a  are  absolutely  mtegrabls,  theaa 
transforms  can  be  defined  as  Fourier  transforms,  with  s(  e  )w^.  and  as  such  they  will 
always  be  well  defined.  A* tentatively,  U  the  system  la  physically  realiaablt,  they  can 
be  defined  as  LopLtce  transforms,  with  s^  *  0.  If  the  system  transforms  srs  known,  the 
system  functions  can  be  determined  (at  least  theoretically). 

The  scoo  if  this  formula  will  be  extended  in  two  way*.  First,  we  form  a  power 
i  erie«  by  allowing  an  infinity  of  terms;  second,  we  relax  the  condition  of  Lebesgue  Inte- 
grability  to  a  condition  of  absolute  integrability  that  permits  impulses  in  the  aystem  func¬ 
tions.  By  using  impulses,  we  can  approximate  some  no-memory  systems,  even  though 
such  systems  are  never  continuoua(R). 

Having  extended  the  scope  of  the  formula,  we  must  impose  some  restrictions  on  tht 
system  functions  in  order  to  guarantee  that  the  formula  will  specify  a  unique  output  for 
every  input  in  a  well-defined  set.  We  define  the  norms  of  the  system  functions  by 


lh„I  ■  f* ...  r  IV’l . vl  <,Tr-*dTn  <«> 

(In  particular,  §t»o g  ■  jh^J ).  We  then  require  that  all  these  norms  be  finite  and  that  the 
power  series 

« 

bh«-£im*  s  i,5> 

n*0 


have  a  nonxero  radius  of  convergence  rh.  Then,  if  the  Input  is  boun.led(R)  for  any  R 
less  than  R^.  all  the  Integrals  will  converge  absolutely,  the  series  will  converge  abso¬ 
lutely,  and  the  output  will  be  bounUed(BH(R));  that  la,  the  value  of  the  output  will  never 
have  magnitude  greater  than  B^(R).  The  function  8^  will  be  i  .Hod  the  bourn.'  function 
of  the  system  H,  and  R^  will  be  celled  the  radius  of  convergence  of  the  system.  A  sys¬ 
tem  that  can  be  exactly  represented  in  this  form,  with  those  conditions,  will  be  caUad 
an  analytic  system. 

The  system  functions  of  an  analytic  ays  em  are  not  unique;  in  tael,  any  h^  can  be 
altered  by  permuting  Its  arguments  T|»  •••»*-,  without  altering  the  system.  However, 

It  can  be  shown  that  unique  system  functions  ars  obtained  under  the  condition  that  the 
eystem  functions  be  symmetric,  l.e.,  that  they  be  unaltered  by  any  permutation  of  their 
arguments.  If  e  system  is  specified  with  uneymmetnc  system  functions,  we  can  replace 
each  system  function  by  the  arithmetic  mean  of  the  functions  obtained  by  all  possible 
permutations  of  its  arguments,  and  the  resulting  system  functions  wilt  be  symmetric 
and  will  define  the  earn*  aystsm. 

U  can  be  shown,  by  using  the  concept  of  multilinear  functions,  that  the  analytic  sys¬ 
tem  is  a  generalisation  of  the  power  series  [cf.  Hille  (Ml).  A  function  f  of  n  arguments 
is  called  a  symmetric  n-linear  function  if  it  is  symmetric  and  tf  tl  is  tinesr  in  each  argu¬ 
ment.  We  ere  intereated  in  two  caaes,  the  familiar  cast  in  which  the  argument#  are 


Any  n- linear 


real  numbers,  xn-i  the  caae  In  which  the  argument*  are  real  Miction*. 

(unction  of  n  real  numbera  can  be  written  in  the  form 

I  Z  II 

-  A*,*.  ...  *.  04) 

i  n  I  <  n 

where  A  la  a  fixed  number.  The  function  1^,  defined  by  a  «  Hftff ....  f  1.  where 
g,  f( . fn  are  alt  real  function*  and 

*'»  •£'”  fl  hn<’* . St  "  V  4Tl*"dTn  ‘W> 

where  h  1*  a  real-valued  symmetric  function  of  n  real  variables,  la  also  a  symmetric 
*"  th 

n-llncar  function.  Now  define  a  homogcner  o  function  of  n  degree  as  a  function  of  a 

single  argument  tli.t  can  be  derived  tro»t  a  wymineti  iC  n- linear  function  by  setting  all 

the  arguments  equal-.  x(  »  x^  *  . . .  «  x^  •  x,  or  f  j  ■  (^  *  . . .  •  f  •  f;  define  a  power 

series  us  an  infinite  summation  of  homogeneous  functions.  With  real-number  argument*. 

we  obtain  lh<  familiar  power  senes;  from  the  n-llncar  function  with  rial-function  argu- 

m  >■'  we  derive  the  formula  that  we  have  used  to  represent  analytic  systems. 

4.2  EXAMPLES 

Two  kinds  o.  examples  are  to  be  described  here;  general  analytic  systems  with  par¬ 
ticular  kinds  of  inputs,  and  particular  kinds  of  analytic  systems  considered  without 
restriction  on  the  input  (except  boundedness). 

in  the  atudy  of  linear  systems,  particular  e  phaals  is  placed  on  two  kinds  of  Inputs; 
Impulses,  which  place  in  evidence  the  physical  significance  of  the  impulse  response, 
and  sinusoids  (or  exponentials).  Which  perform  a  like  service  for  the  transform  of  the 
impulse  response.  Sums  of  Impulses,  or  sums  of  sinusoids,  are  trivially  accounted 
for  by  superposition;  but  superposition  does  not  hold  in  nonlinear  systems. 

Consider  an  analytic  system  whose  input  In  a  sum  of  imj—’se*.  Strictly,  UUa  is  not 
pertnisuible,  because  a  well-defined  output  is  guaranteed  only  for  bounded  Inputs.  How. 
ever,  a  formal  result  can  be  obtained,  and,  since  laboratory  approximations  to  impulses 
are  really  bounded  pulses,  the  formal  result  has  a  practical  interpretation.  For  the 
sake  of  simplicity,  we  consider  an  analytic  system  with  only  one  term,  l,e.,  all  the  sys¬ 
tem  functions  except  one  are  sero;  a  general  analytic  system  is  a  aum-of  such  systems. 
Let  the  input  f  to  such  a  system  br 

M 

f(t).  J  Am*(t-Tm)  (,*, 

m>  I 

that  is,  a  sum  of  M  impulses,  with  the  m*^  impulse  having  the  value  Am  and 
occurring  at  time  Tm.  Then  the  output  g  of  the  system  ie  given  by 
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Each  permutation  of  every  combination  of  n  impulse*  (not  necessarily  a  different 
Impulses)  chosen  from  the  input  thus  fives  a  contribution  to  the  output;  this  output  is 
equal  to  the  product  of  the  values  of  these  Impulses  with  an  appropriate  value  of  the  oys. 
tern  function.  The  integrals  that  appear  in  the  general  expression  for  the  output  have 
been,  in  effect,  replaced  by  sums.  If  we  consider  a  general  input  aa  approximate  by 
a  sum  of  impulses,  then  the  output  at  any  time  is  a  weighted  sum  of  products  of  past 
values  of  the  input,  the  weighting  being  determined  by  the  system  functions.  The  system 
functions  of  an  analytic  system  might  therefore  be  considered  as  ita  impulse  response 
functions. 

Now  suppose  the  input  is  a  sum  of  sinusoids,  or,  since  a  sinusoid  can  be  expressed 
as  a  sum  of  complex  exponentials,  suppose  the  input  is  a  sum  of  exponentials.  Thus 


M 

W>  -  y  Am  "PlV*  <3»> 

m*l 


Then  the  output  g  of  the  system  is  given  by 


M 

g(t).  £  . 
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Each  permutation  of  every  combination  of  r.  exponentials  (not  necessarily  n  different 
exponentials)  chosen  from  the  Input  gives  a  contribution  to  the  output  that  ia  an  exponen. 
tial  with  a  complex  frequency  equal  to  the  sum  ol  .he  frequencies  of  these  exponentials, 
and  wi  an  amplitude  that  is  equal  to  the  product  of  the  amplitudes  of  these  exponentials 
and  an  appropriate  valus  of  the  system  transform.  Since  a  sinuso.d  is  a  sum  of  two 
complex  exponentials,  each  with  frequency  equal  to  the  negative  of  the  other,  these 
contributions  account  for  the  harmonics,  sum  frequencies,  and  difference  frequencies 
that  we  know  occur  in  nonlinear  systems.  The  system  transform  gives,  in  terms  of  the 
magnitudes  and  phases  of  the  input  sinusoids,  the  magnitude  and  phase  of  esch  sinusoid 
in  the  output.  The  system  trsnstorres  might  therefore  be  considered  as  frequency* 
response  functions. 

We  now  consider  some  special  types  of  analytic  systems.  First,  we  consider  the 
identity  system,  whose  output  always  equals  its  input.  Wc  shall  represent  this  system 
by  X.  This  is  a  linear  no. memory  system.  AU  its  system  functions  are  zero  escept 
the  first,  1|,  which  is  s  unit  impulse;  hence,  all  its  system  transforms  ere  zero  except 
the  first,  Ij(e)  ■  1. 
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Next,  we  consider  two  type*  of  systems  that  are  eaay  to  deal  with  by  tnethoda  that 
are  already  in  wide  use:  linear  ayatema  and  no-memory  eyatema.  A  linear  ayatem  ta 
analytic,  with  infinite  radiua  of  convergence,  U  ita  impulse  response  la  absolutely  irile- 
grable;  thia  impulse  reeponae  la  then  Ita  flrat  ayatem  function,  and  all  the  other  ayatem 
functiona  are  aero.  The  flrtt  ayatem  transform  la  the  frequency-response  function  of 
the  linear  system,  and  all  other  ayatem  transforms  are  aero.  Conversely,  an  analytic 
system  le  linear  tf  all  ita  ayatem  functiona  (or  ayatem  transforms)  are  aero  except  the 
first, 

A  no-memcry  system  la  analytic  tf  the  value  of  the  output  la  given  in  terms  of  th« 
value  of  the  input  by  a  power  series,  «nd  its  radius  of  convergence  is  the  radius  of  con¬ 
vergence  of  this  power  series.  U 

m 

B(‘>  ■  £  »ft(t(t))n  (40) 

n«0 

then  the  system  functions  sre  all  products  of  impulses, 

Vtr-:*,n|,V<,l)“Tl|-  S<Tn>  («»> 


which  indicates  that  the  value  of  the  output  at  any  time  ia  independent  of  past  values  of 
the  input.  Tht  system  transforma  are  all  constants. 


Hn<V" 


■  V*n 
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which  indicates  that  if  the  Input  la  a  sum  of  sinusoids,  the  amplitude  and  phase  of  the 
output  sinusoids  are  independent  of  the  frequencies  of  the  input  sinusoids.  Cunversely, 
if  the  system  transforms  of  an  analytic  system  arc  all  constants,  the  system  is  a  no. 
memory  ayatem. 

4.J  COMBINATIONS  OF  ANALYTIC  SYSTEMS 


B<  .ause  engineering,  at  the  practical  level,  consists  largely  of  putting  things  together 
and  making  them  work,  analysis  and  synthesis  have  become  important  parts  of  the  theory 
of  linear  systems,  and  they  may  be  expected  to  bn  important  in  the  theory  of  nonlinear 
systems  as  well.  Analysis  is  generally  easier  than  synthesis,  and  it  may  be  that  the 
best  way  to  develop  a  good  theory  of  synthesis  is  \o  develop  first  a  good  theory  of  analy. 
sis.  An  approach  to  the  analysis  of  nonlinear  systems  is  proposed  In  Mils  section  und 
elaborated  in  subsequent  sections. 

In  the  fundamental  approach  we  begin  with  analytic  ayatema  und  interconnect  them  In 
several  ways,  with  the  object  of  determining  when  the  result  of  this  interconnection  con¬ 
stitutes  an  analytic  system,  and,  whenever  it  does,  what  it j  system  functions  or  system 
transforms  are.  Many  practical  systems  can  be  described  as  combinations  of  linear  and 
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no-memory  systems,  which  can  be  e  aaily  represented  ui  analytic  form.  For  each  ays- 
terns,  this  approach  may  be,  tor  some  purposes,  an  adequate  method  of  analysis.  We 
begin  with  some  simple  forms  of  interconnection:  sums,  products,  and  cascade  com* 
bmationj. 

The  sum  |i  ♦  of  two  systems  |1  and  K  is  constructed  by  tying  their  inputs  together, 
so  that  the  same  inpul  is  applied  to  both,  and  adding  their  outputs.  Thus,  9  •  H  ♦  K  il 
and  only  if,  for  every  input  f,  §10  ■  H(0  ♦  K{f). 

If  the  eysteme  H  and  &  are  analytic,  a  trivial  calculation  gives  the  result  that  the 
system  functions  of  §  are 
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and  the  system  transforms  of  §  are 
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But  these  results  are  not  sufficient  to  show  that  §  is  analytic,  since  we  st...  have  to 
show  the  existence  of  a  bound  function  and  a  nonzero  radius  of  convei  „cnce.  However, 
this  is  not  difficult.  It  follows  from  Eq.  41  that 


K1 « bj *  «M 

and  from  this  bound  on  the  norms  of  the  sv  .  . 
upper  bound  on  die  bound  functlrei,  ‘ 

Bq(x)  «  Bh«x)  ♦  BK(X) 

and  a  tower  bound  on  the  radius  of  convergence, 
Rq  *  min  IRh,  Rk> 
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nctiona  of  §  we  can  determins  an 
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Note  that  if  is  a  sy  stem  of  degree  N  U.e.,  all  system  functions  after  the  N  are 
aero),  and  1£  is  a  system  of  degree  M.  then  the  degree  of  §  will  not  exceed  the  larger 
of  the  two  numbers  N,  M.  , 

Almost  as  eerily  tre.-ted  is  the  product  HK  of  n  o  systems,  constructed  by  tying  the 
inputs  together  and  multiplying  the  outputs:  §(0  ■  H(0  K<0.  If  H  and  K  are  analytic,  s 
straightforward  calculation  of  the  output  Of  their  product,  after  terms  of  like  degree  are 
collected,  yields 
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From  this,  wf  determine  del 


KJ‘Z  lhJ  *K-»I 


from  which  It  follow* 


bQU)  -  BhU)  *»Kt*> 


RQ  >  min  (52) 

If  U  ia  a  system  of  degree  N,  and  la  a  system  of  degree  M.  then  the  degree  of  § 
will  not  exceed  N  i  M. 

I  he  cascade  combination  R  •  K  of  two  irittmi  H  and  K  la  formed  by  applying  the 
input  to  K,  using  the  output  of  K  as  the  input  to  H,  and  taking  the  output  from  H.  Then 
Q  <  H  •  K  it  and  only  U.  lor  every  input  f.  §(0  »  H(K(f)).  Note  that  H  •  K  and  K  •  Hate 
not  the  same,  although  in  special  cases  (e.g..  linear  systems)  they  are  equivalent. 

The  system  fur*  lions  and  system  transforms  of  Q  are  given  by  formulas  that  are 
derived,  as  In  the  product  case,  by  a  straightforward  computation  of  the  output.  In  which 
terms  of  like  deg'ee  are  then  collected.  These  formulas  are  rather  complicated  in  the 
general  case,  although  they  can  be  expressed  fairly  simply  in  certain  almost  general 
caaea. 

The  first  atep  in  the  calculation  of  ^  or  ^  is  to  determine,  for  every  positive  inte¬ 
ger  i,  all  possible  permutation*  of  combinations  of  i  non-negative  integers  whose  sum 
is  n.  In  each  permutation  these  i  integer*  will  b>’  called  Dt^j>l,2,...,i.  The  system 
function  qft  Is  given  by  a  ■Lonvolution-like  integral  involving  hn  and  km  .k^ . hm  : 

*  r"  r"  • 

*n<Tl . Tn 1  *X/lJ  *•*/  hiixl . *t*  n •.><!*,•  ••d»t 

1*0  •-«  »-»  *  * 


The  order  of  the  subscript  indices  on  the  r*s  is  immaterial,  because  permutation  of  the 
arguments  of  a  system  function  does  not  change  the  system  It  characterizes;  hence  they 
are  not  indicated.  The  only  important  point  to  remember  is  thit  there  should  be  one  of 
each,  from  t  through  a.  and  the  t*s  might  as  wel.'  be  indezed  in  the  order  in  which  they 
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appear  when  the  km  are  written  in  order  of  increasing  ),  The  mtmd  summation  azgw 

In  this  formula  indicates  a  summation  over  the  set  of  permutations  indicated  at  the 
beginning  of  the  paragraph. 

The  system  transforms  ^  are  given  by 
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«  is  '.he  sum  of  the  m  argument*  of  K_ 
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In  this  formula,  aa  in  formula  5  J,  the 


second  summation  is  over  all  permutations  of  i  numbers  whose  sum  is  n,  and  the 
a-varUbles  may  be  indexed  l,,..,n  in  any  order. 

In  the  general  case  an  infinite  surrmalion  is  required  for  the  determination  of  each 
system  transform  or  system  function.  However,  when  i  is  greater  than  n,  every  com* 
binaUon  of  ci^  must  contain  at  least  one  that  is  equal  to  xero.  Therefore,  there  are 
two  special  cases  in  which  each  r  ^  can  be  determined  aa  a  sum  of  a  finite  number  of 
terms:  when  H  is  a  system  of  finite  degree,  and  when  KQ  ■  0. 

In  many  practical  systems  it  is  possible  to  obtain  a  description  m  which  the  compo¬ 
nent*  are  analytic  systems  with  aero  constant  terms,  by  first  solving  for  the  particular 
case  of  zero  input  to  determine  the  operating  points,  and  then  measuring  »U  variables 
with  respect  to  these  operating  points.  If  both  systems  in  a  cascade  combination  hare 
zero  constant  terms  (l.  e. ,  HQ  * 
cascade  combination  wc  obtain 


KQ  «  0J,  then  for  the  first  few  system  transforms  of  the 
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Q,(s)  .  H,(s)  K,(s) 

Q,(»[.s2)  «  »,(«,  ia2)K2(a,.a2)  ♦  HjUj.s.lKjUpK,^)  (5S) 

In  analytic  systems,  as  In  linear  systems,  the  solution  of  the  cascade  problem 
involves  integration  in  the  time  domain,  but  dot  a  not  involve  integration  if  the  solution 
is  expressed  In  the  frequency  domain. 

To  determine  bounds  on  norms,  bound  function,  and  radius  of  convergence,  in  the 
general  case,  we  obtain 
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from  which  it  car  be  shown  that 
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•>e  resultant  »y»iem  Q  will  not  always  be  vojjta,  there  might  not  be  any  nontero 
ra.'iu*  of  convergence  if  Kq  ia  greater  than  R^,  However,  the  same  condition*  that 
enaw?ed  that  the  system  t unctions  and  system  transform*  of  ^  could  be  computed  by  a 
finite  summation  will  also  ensure  the  analytictty  of 

If  the  components  of  a  cascade  combination  are  of  degree  N  and  M,  respectively, 
the  degree  of  the  resultant  system  will  not  exceed  Kfl. 

In  aU  cases,  sum.  product,  and  cascade,  the  bounds  on  the  bound  function  are 
immediately  obvious  if  they  are  regarded  only  as  bounds  on  the  output.  However,  the 
bounds  cn  the  bound  functions  provide  more  information  than  this.  The  coefficients  of 
the  power-series  expansion  of  the  bound  provide  upper  bounds  on  the  norms  of  the  sys¬ 
tem  functions.  Furthermore,  the  formulas  for  bounds  on  the  bound  functions  and  radii 
of  convergence  remain  valid  if  the  bound  functions  sod  radii  of  convergence  that  appear 
u.  the  form-das  are  replaced  by  bounds  that  have  been  obtained  from  a  previous  calcu¬ 
lation. 

The  bound  (unctions  of  combination  systems  do  not  have  to  be  computed  by  power- 
senes  methods;  they  may  be  computed  graphically  or  numerically.  Because  the  system 
calculations  mu.t  be  performed  by  calculating  the  system  functions  or  transforms  ons 
st  a  Ume.  the  bound  functions  provide  a  useful  for  controlling  their  accurac- 

If,  for  example,  only  the  first  three  system  ■  '.-L  c.  i  a  combination  system  are  cal¬ 
culated.  a  bound  on  the  error  can  be  obtained  by  .  Mracting  from  the  upper  bound  of 
its  bound  function  the  first  three  terms  of  it*  power- series  expansion.  The  resulting 
function  gives,  for  every  R,  ?n  upper  bound  on  the  magnitude  of  the  error  for  inpjts 
that  are  boundedfRJ. 


4.4  EXAMPLES  OF  CASCADE  SYSTEMS 


It  may  often  happen  in  practice  that  only  one  system  in  a  cascade  chain  is  nonlinear, 
and  *11  the  other*  are  linear.  In  such  a  chain  some  fairly  simple  and  easily  rrcognixabl* 
forms  are  obtained  for  the  solution. 

For  a  Unear  oystt  -  V  followed  by  a  nonlinear  analytic  system  jl  (i.e.,  for  the 
combination  Q  »  H  •  L),  we  obtain 

Si<si . v •  *V*1 . V  L1U») Litv  •••  h’V  ,y,) 


For  a  nonlinear  *yatem  followed  by  •  linear  system  (i.e.,  for  the  combination 
S  *  k  *  H).  *t  obtain 

^(s, . aB)  »  LjUj**^...  +*a)  Hn.*, . sBJ  (601 
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For  a  nonlinear  system  U  preceded  by  a  linear  ayaiem  ^  and  followed  by  a  linear 
ay  item  K  fi.e.,  for  ‘he  combination  $  «  K  a  H  •  L),  we  obtain 

«n<*l . „>  “  *,<'!♦•“  ♦  V  HnC, . •„>  Sf.,1  S‘V  «•» 

In  the  particular  caae  In  which  the  nonlinear  ayatem  laa  no. memory  ayatem,  ao  that 

each  ayatem  transform  H  la  a  constant  A  .  wa  have 
n  n 

QJ*, . •„)  •  AnK,(eI»...ea|l)  L^a,)...  L,(an)  (62) 

This  form  is  ao  easy  to  recognize  that  it  can  be  uaed  for  ryntheale,  since  any 
analytic  system  whose  system  transforms  are  of  thie  form  must  be  a  cascade 
combination  of  linear  systems  with  one  no.memory  ayatem,  or  the  equivalent  of 
such  a  combination. 

As  an  illustration  of  the  solution  of  cascade  combination  aystems,  we  consider 
an  amplitude-modulated  radio  communication  system.  We  ahall  suppose  that  the 
carrier  and  the  signal  are  added  in  the  transmitter,  and  paaa  through  a  nonlinear 
no-memory  device  that  acta  aa  a  modulat.  .  "  he  output  of  the  modulator  passes 

through  a  radiofrequency  amplifier,  a  propagation  path,  and  several  radiofrequency 
ainplifierc  in  Jie  receiver,  ail  of  which  are  represented  by  a  linear  narrow. band 
filter.  Another  nonlinear  no-memory  device  acta  as  a  detector,  and  the  output  of  the 
detector  then  paeecs  through  a  sequence  of  audio-frequency  devices,  represented 
by  a  linear  filter. 

The  modulator  will  be  assumed  to  be  a  second.degree  system,  whose  output  y  in 
terms  of  its  input  x  is  given  by 

2 

y  s  rrtjX  +  rrt^x  (61) 

The  radiofrequency  channel  will  be  aesumed  to  be  linear  with  frequency. response  func¬ 
tion  K(s).  This  filter  will  be  assumed  tc  have  zero  response  st  audio  frequencies  and 
at  the  harmonica  of  the  carrier.  The  detector  ie  another  second-degree  device,  with 
output  y  in  terms  of  input  x  given  by 

y  »  dj*  ♦  d 2*2  (64) 

The  audio-frequency  channel  will  be  assumed  to  be  linear  with  frequency. response  func¬ 
tion  A(e),  and  the  rraponec  of  this  channel  will  be  assumed  to  be  zero  *i  zero  frequency 
and  at  radio  frequencies. 

The  fundamental  cascade  combination  formula  ia  then  applied  three  times  in  succes* 
aion  to  obtain  the  system  transforms  o'  the  complete  channel: 

» 


Qo.0 


Qz(«|.*2>  •  ♦a2)R(»J  ♦•j)  ♦  ♦  a^Rfs^RISg) 

*j)  •  mjin^A^j  ♦•2»*jJR(«1>R(*2»»j) 

♦  m|mzA(a,  ♦  »2  +  #,)R(a,  f  •2»R(»J) 

Q4(a,.....«4)  *  fn2dzA(«(  4  B2  +  a)  +  «4>R<*|  +*2)R{*j*»4>  (t5> 

and  all  further  system  tranaf^-ma  are  taro.  However,  Q(  and  the  firat  term  of  are 
eero,  since  we  have  aaaumed  that  A|s)R(s)sO,  and  the  two  terms  of  Q}  can  be  combined 
by  permuting  the  variables  In  one  of  them.  The  only  nonzero  system  transforms  then 
become 

Q2(«,,*2)  ■  mjdjAfSj  ♦e2»R<*l>’..<*2) 

Qj(s|,s2,sj»  ■  Zm^^Ali,  ♦s2esJ)R(s1)R(s2  +  SJ) 

Q4(e,. ..  ..a4)  «  m“d2A(Sj  ♦  s2eSj  +  s4)R(s,  ►  s2)  R(s  j  4  «4)  (66) 

These  transforms  characterize  the  complete  channel  as  a  nonlinear  analytic  system. 

Now  auppone  that  the  input  consists  of  a  number  of  sinusoids,  one  at  the  carrier 
(radio)  frequency,  with  exponential  components  at  a  ■  and  the  rest  at  audio  fre. 

quencles,  a  ■  ajiu^,  ajwa2.  an<*  *°  forth.  Then  no  output  will  be  obtained  from  Q2, 
since  the  only  frequencies  in  the  input  for  which  R(s)  is  nonzero  t  re  the  positive  and 
negative  carrier  frequencies,  whose  sum  is  either  zero  or  twice  the  carrier  frequency, 
for  both  of  which  A(s(  ♦  s2)  1*  zero. 

A  nonzero  output  is  obtained  from  only  when  S|  is  a  carrier  frequency  and  a2  +  a} 
is  the  sum  of  a  carrier  and  an  audio  component,  and  the  two  carrier  frequencies  are  of 
opposite  sign;  the  sum  frequency  will  then  be  the  audio  frequency.  (For  each  audio  com¬ 
ponent  there  will  be  four  terms:  s(  either  plus  or  minus,  and  w  'h  either  J,  or  * ^  as 
the  audio  component.)  Therefore  gives  the  demodulated  audio  output,  which  is  pro¬ 
portional  to  the  audio  Input,  the  square  of  the  carrier- frequency  Input,  the  audio, 
frequency  gair,  the  square  of  the  radiofrequency  gain,  the  linear  part  of  the  modulator, 
the  eecond-degree  part  of  the  modulator,  and  the  tecond-degree  part  of  the  dotector. 

Q4  gives  s  nonzero  output  only  if  both  Sj  ♦  s?  and  a}  +  s4  are  nimi  of  audio-  and  csrrrr. 
frequency  components,  and  the  sum  frequency  le  the  sum  of  the  two  audio  frequencies. 
Hence  gives  the  harmonic  and  utter  modulation  distortion  components  in  the  output. 

These  particular  results  could  also  havs  been  obtained  by  more  conventional  methods 
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by  assure. ng  the  Input  and  then  calculating  the  resulting  signals  at  every  point  In  the 
system.  What  we  have  done  here  la  to  solve  the  ayatea  aa  a  system  before  specifying 
what  the  input  is  to  be.  — 


4.  S  ANALYTIC  FEEDBACK  LOOPS 

An  electrical  network  consists  of  a  number  of  elements  connected  ->y  wires. 

In  many  cases,  the  elements  arc  all  two-terminal  elements,  and  each  element 
con  be  deecribed  as  a  system  with  one  Input  and  one  output,  by  specifying 
either  the  voltage  in  terms  of  the  current  or  the  currant  in  terms  of  the  volt* 
age.  The  interconnections  are  expressed  in  terms  of  Kirchhotfs  laws,  which 
equate  to  eero  either  a  sure  of  voltages  or  a  sum  of  currents.  These  rela¬ 
tions  can  be  expressed  by  a  block  diagram  or  signal-flow  graph  that  contains 
two  kinds  of  components:  systems  and  summing  points.  It  appears,  therefore, 
that  a  theory  of  nonlinear  network  analysis  might  be  built  up  by  uaing  only 
two  of  the  three  kind*  of  simple  combination  described  in  eeclion  i.  3,  namely, 
sums  and  cascade  combinations. 

Such  a  theory  has  not  yet  been  developed.  In  terms  of  the  relation  between 
aystema  and  functions,  as  developed  in  Section  1,  this  theory  would  be  essen¬ 
tially  a  theory  of  Implicit  systems.  We  might  hope  for  an  extension  to  ays. 

tern  theory  of  the  fundamental  theorem 
{cf.  Rudin  (15)]  can  Implicit  functions, 

For  the  yjrpc~e*  of  suggesting  the  kind 
of  results  that  such  a  theory  might  offer, 
smd  of  giving  a  special  case  with  its 
own  useful  applications,  the  solution  of 
additive  feedback  loops  with  analytic 
components  is  presented  in  this  section. 
Another  special  case,  the  Inverse  of 
an  analytic  system,  will  be  discussed 
in  section  4.7. 

Consider  the  simple  feedback  loop 
Illustrated  In  Fig.  3.  This  le  not  such 
a  special  case  (as  it  seems)  because, 
ss  Fig.  4  shows,  s  general  feedback 
loop  can  be  reduced  to  two  cascade 
problems  and  to  a  feedback  problem 
of  the  simple  form. 

Assume  that  this  simple  loop  sys¬ 
tem  is  equivalent  to  a  system  &.  Let 
the  input  be  f,  and  the  output  g  •  &(f); 


Fig.  3.  Simple  feedback  loop. 


Fig.  4.  Reduction  of  general  loop  to 
simple  form. 
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then 


K«0  •  f  *  iHtS(O)  (*T> 

or,  ••  •  system  equation  with  the  notation  of  section  4.  ), 

K  ■  I  *  tt  •  &  (*»» 

where  1  designates  the  identity  system. 

Now  suppose  that  H  is  an  analytic  system.  We  assume  that  K  is  analytic,  and 

obtain  a  formal  solution  -  we  shall  determine  later  whether  K  is.  in  fact,  analytic.  We 

stall  consider  that  Hc  ■  0,  since  a  nonzero  HQ  represents  merely  a  constant  added  to 

the  input,  and  we  do  not  have  to  consider  this  as  part  of  the  feedback  loop.  Then,  for 

K  .  we  obtain 
o 

Kq  •  II  j(0»  K0  ♦  1I2(C,  0)  K*  ♦  Hj(0,  0.  0)  K*  ♦  .  . .  (69) 

This  equation  may  have  many  solutions,  but  it  will  always  have  the  solution  Kq  *  0.  We 
assume,  as  is  often  me  esae  in  practice,  mat  we  see  looking  for  the  solution  thet  gives 
zero  output  for  zero  input,  and  hence  we  accept  the  solution  Kq  ■  0.  This  allows  us  to 
use  the  simplified  form  of  the  cascade  equations  which  occurs  when  the  constant  ten..* 
of  both  components  are  zero. 

The  next  equation  that  we  obtain  is 

K,(s)  •  1  ♦  H|<»)  K|(s)  (70f 

from  which  we  determine  that 

Kl<“)  *  1  -  H,(s)  f7|> 

This  is  the  result  that  would  be  obtained  from  an  approximate  linear  analysis.  Next  we 
obtain 

K2(st,s2)  -  H^SjfSjJKjlSj.Sj)  f  H2(s|.s2)K|(s|)K,(s2)  (72) 

and,  since  K|  has  already  been  computed,  thia  equation  can  be  solved  for  K^,  and  we 
have 

-w*. 

H2(Si#s2) 

K2<*«'  *2>  "  (1  -  H,(S|  ♦  s2>H»  -  Hjls^Hl  -  H,(s2))  17 W 

This  procedure  can  be  continued  indefinitely,  since  all  the  formulas  for  higher  order 
Bystems  transforms  Kn,  as  derived  directly  from  the  cascade  formuls,  will  contain 
Kfl  only  in  one  term  on  the  right-hand  side.  Furthermore,  since  Kn(Sj, .  appears 

on  the  right,  multiplied  only  by  Hj(*j  ♦  ♦  sn),  the  only  factors  in  the  denominator 
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Fig.  5.  Modification  of  loop  for  proof  of  analytlcity. 
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of  the  resulting  expression  for  Kfl  will  be  cf  the  form  1  -  II  ,  with  various  arguments. 

The  form  of  the  solution  suggests  that  the  stability  of  the  resultant  system  j<,  and 
its  analyuctty  ss  well,  might  be  determined  solely  from  the  linear  approximation.  This 
is  a  conclusion  that  has  been  reached.  In  a  somewhat  different  context,  by  Polncarf 
and  L-iapounoff.  and  It  will  now  be  shown  that  it  is  essentially  true  also  in  this  case. 

We  begin  by  separating  the  system  U  into  two  parts,  a  linear  part  ft’,  and  a  part 
H*  containing  no  lii.ear  term,  as  shown  tn  Fig.  i.  We  then  solve  the  linear  part  of  the 
feedback  loop  separately,  and  obtain  a  loop  with  a  forward  path  through  the  linear  sys¬ 
tem  K1  which  is  equal  to  the  linear  part  of  K,  and  a  return  path  through  H* ,  The  solu¬ 
tion  of  this  loop  is  the  same  as  the  solution  just  obtained;  however,  we  are  not  interested 
now  tn  the  formal  solution  for  the  system  transforms,  but  rather  In  obtaining  bounds  on 
the  bound  function  and  the  radius  of  convergence. 

For  this  modified  U  r  obtain,  for  an  arbitrary  input  f, 

K(f)  .  &'[f  ♦  H"(l£tl))]  IU) 

or,  in  terms  of  systems,  we  have 

U  •  K*  •  U  ♦  U*  •  K)  <7» 

Since  ]£'  Is  Linear,  by  superposition  Eq.  74  becomes 


K  »  S’  ♦  &’  •  U*  •  & 

The  known  system  •  jj*  will  now  be  designated  by  £.  Then 
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Therefor*. 

BKf*)  *  UK,Ul  ♦  Bq(Bk(>|) 

«lkil*  ♦bq<bku»  m> 

The  cunaunt  and  Unear  terms  of  the  power. series  expansion  of  Bq  are  both  sero,  so 
that  dus  equation  can  be  used  directly  to  determine,  one  at  a  time,  upper  bounds  on  the 
coefficients  of  the  power.serlcs  expansion  of  B^:  that  is,  bounds  on  the  norms  of  the 
system  functions  kn<  This  procedure  is  a  solution.  In  power. series  form,  of 

y  •  fk|I*  ♦  Bq<yi  (**) 

and  yix)  is  an  upper  bound  on  B^(x). 

A  solution  can  be  obtained  without  using  power  series  bv  solving  this  equation  for  z 
in  terms  of  y  and  graphing  the  result,  ss  follows 

*  »  O'  -  (BO) 

As  ;  increases  from  sero,  x  Increases  from  sero.  reaches  s  maximum,  and  thereafter 
decreases,  unless  the  radius  of  convergence  of  Bq  (which  is  equal  to  Rjj)  is  so  small 
that  the  curve  comes  to  an  end  before  the  maximum  value  is  reached.  Equation  BO  gives 
x  as  an  analytic  function  of  y,  and  investigation  of  the  analytic  cooUnualion  of  this  fuoc~ 
uon  in  the  complex  plane  indicates  that  its  inverse  is  analytic  in  a  circle  about  the  origin 
with  radius  equal  to  the  maximum  value  01  s. 

Hence  the  Inverse  of  this  function  —  which  can  be  obtained  immediately  from  the 
graph  —  is  the  desired  upper  bound  on  the  bound  function  B^.  The  maximum  value  of 
s,  which  will  occur  either  at  the  end  of  the  curve  (ss  determined  by  R^)  or  at  the  turning 
point,  is  a  lower  bound  oi>  R^.  Thus  the  resultant  system  It  can  be  proved  analytic  if 
Jkg|  exists;  dial  is,  if  the  linear  approximate  solution  of  the  feedback  loop  has  an  abso¬ 
lutely  mtegrsble  impulse  response. 

Note  that  the  impulse  response  can  he  absolutely  integrable  only  if  the  linear 
approximation  does  not  include  a  differential  operator  and  has  only  damped  tran¬ 
sients,  Stability  is  therefore  a  necessary,  but  not  quite  sufficient,  condition  for 
analyttcity. 

4.4  EXAMPLE  OF  FEEDBACK  SOLUTION 

Section  4.  $  constituted  essentially  a  proof  of  the  existence  of  the  solution  to  certain 
problems  and  an  outline  of  the  me  mod  for  their  solution.  In  this  section  the  method  is 
applied  to  a  specific  problem,  with  two  changes  in  the  method.  First,  the  feedback  loop 
is  solved  without  first  reducin'  it  to  the  simple  form  treated  in  section  4.4.  Second, 
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the  result  alii  b  proved  analytic  in  spite  of  the  fact  that  one  of  its  components  Is  not 
analytic. 

The  system  to  be  analysed  is  the  detector  circuit  of  Pig,  be.  which  consists  of  a 
diode  and  a  capacitor.  Since  the  diode  is  not  pei  feet,  no  resistor  ts  needed  in  parallel 
with  the  capacitor.  The  diode  ie  assumed  to  have  the  current. voltage  relation 

i  ■  AJeBT  -  I)  (St) 

and  the  capacitor  has  capacitance  C. 

The  relations  governing  the  operation  of  the  circuit  can  be  expressed  in  the  block 
diagram  of  Fig.  6b.  The  system  fi  is  a  no.tnemory  system  representing  tha  diode 
with  voltage  input  and  current  output;  Dq  ■  0  and 

DnU, . •„>  -  AH"/"'  (32) 

for  ail  other  n.  The  system  £  is  linear,  representing  the  capacitor  with  current  input 
and  voltage  output;  Cj(s)  a  l/Cs.  The  system  £  is  not  analytic. 


-» 
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r'ig.  6.  Detector  circuit,  (a)  Circuit  diagram. 

(b)  Blots  diagram,  (e)  Synthesis  of  tha 
approximate  solution. 
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n  obtain 


Letting  H  •  £  •  J>,  by  a  ample  application  of  the  UMtduj  It 


H  ■  0  and 

u 


..  ) - *S! - 

n  nICISj*...  ssR) 


Now,  if  we  represent  the  entire  detector  system  by  £,  we  obtain  the  wywaegt  equation 
fr'tm  the  block  diagram. 


£eU«U-J^  (64) 

Thi*  is  not  of  the  Mine  forts  aa  Eq.  66.  which  waa  treated  in  section  4.5.  but  the  same 
method  of  solution  la  applicable.  Ft  r  the  first  few  system  transforms,  n  obtain 

K  «  C 
o 

eirb  ‘*5» 

AB2C2sjs2 

"  :<cis ,  ♦  »2)  ♦  ABMCS ,  ♦  AB)(Caz ♦  AB) 

and  further  system  tranaforma  can  be  calculated  ut  succession. 

The  system  transform  Kj  can  be  recognised  as  having  the  form  characteristic  of  a 
no- memory  »y»> rm  preceded  and  followed  by  linear  syatema,  and  the  aett— 1  -degree 
approximation  to  £  can  thus  be  synthesised  in  the  form  of  Fig.  6c  fresn  tike  transforms 
given  in  Eqs.  65. 

The  solution  given  is  only  s  formal  one,  in  the  sense  that  it  will  be  valid  if  K  Is 
analytic,  but  we  do  not  yet  know  whether  it  ia  analytic.  To  show  snalyUciry,  we  view 
the  circuit  from  a  different  point  of  view.  We  consider  the  system  with  input  C|,  as 
before,  but  with  v  as  the  output,  and  call  this  system  £.  Since  v  »  we  find  that 

E  ■  1  -  &.  Therefore,  K  is  analytic  if  and  only  if  £  is,  and  these  two  systems  have 
the  came  radius  of  convergence.  Furthermore,  the  norms  of  their  symm  functions 
will  be  the  same,  except  for  the  first,  so  that  Bp  can  be  used  to  determine  bounds  an 
the  error  that  results  from  using  only  a  finite  number  of  terms  of  tbe  expression  for  J&. 

The  block  diagram  for  £  has  the  t .tuple  form  of  the  loop  discussed  in  section  4,}, 
Since  £  is  linear,  we  find  £•  and  H*  by  seoaratlng  out  the  linear  part  of  Q.  For  the 
linear  approximation,  we  obtain 

pi<*»-cJTTS  “fc» 


from  which  we  determine  that 

p,(t|  .  4(t)  -  (AB/C)  »-***/&* 


aad  therefore  Jpsfl  ■  t.  Proceeding  at  in  the  previous  section,  we  find  that 
5  »  -F  •  C  •  B*.  and  when  we  cascade  £'  and  £  we  tee  that  the  troublesome  a  in  the 
’  denominator  of  C  j  la  canceled  by  the  numerator  of  P(.  to  that  3  ta  analytic.  From 
the  formula  for  the  upper  round  on  the  bound  function  of  a  caacade  combination,  we  find 

BqU)  « (eB*  -  1  -  Bxl/B  _  (»»> 

The  bound  yfx)  on  the  bound  function  B^(x)  ia  therefore  determined  by 

a  .  ,  -  <eBy  -  l)/ZB  (»»» 

The  maximum  \alue  of  *.  which  ia  a  lower  bound  on  Rp»  R^,  ia  (logZ  -  l/Z)/B  «  O-lfl/B. 
The  graphical  conatruction  of  the  upper  bound  on  B^lx)  ia  ahown  in  Fig.  7. 


I  tg.  7.  Bond  on  the  bound  function  of  the  detector  ijittm. 


This  reault  is  somewhat  disappointing,  In  that  we  would  expect  a  much  larger.  In 
fact  an  infinite,  radius  of  convergence.  The  radius  of  convergence  may  in  fact  be  inft> 
nite,  but  the  method  we  have  used  is  inherently  incapable  of  Indicating  an  infinite  radiue 
of  convergence.  It  la  a  conservative  method,  based  essentially  on  the  assumption  that 
the  feedback  through  the  nonlinear  terms,  which  we  are  unable  to  compute  in  general 
except  to  determine  a  bound  on  its  magnitude,  ia  exerting  the  greatest  possible  effort 
to  make  the  system  unstable.  Further  research  might  reveal  less  conservative  tests 
for  analyttcity.  At  any  rate,  we  have  proved  that  the  radius  of  convergencs  is  greater 
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than  sero,  and  have  therefor* jesubllshed  some  validity  tor  the  result  of  the  system  cal¬ 
culation. 

4.7  THE  IN /ERSE  OF  AN  ANALYTIC  SYSTEM 

If  U  is  s  nonlinear  system,  which  produces  for  every  input  f  in  a  certain  class  a 
corresponding  output  g  •  U((),  the  inverse  H”*  of  {1  is  a  system  which,  tor  every  Input 
g.  produces  an  output  f  such  that  g  *  UtO.  Then  the  cascade  combination  tf  •  fl"*  la 
the  identity  ayetem  1. 

This  dues  not  imply,  in  general,  that  H_1  •  ii  ■  1,  because  if  H  can  produce  the  same 
output  fur  iwo  different  inputs,  jt  1  cannot  tell  which  of  these  inputs  actually  occurred. 
Generally,  we  shall  say  that  if  has  a  true  inverse  only  if 


»•  U”1  •U't  •&*!  (90) 

The  inverse  of  a  physically  realisable  system  is  not  necessarily  physically  realis¬ 
able.  This  is  known  in  the  theory  of  linear  systems;  the  inverse  of  s  simple  delay  ays. 
tern  would  be  a  simple  anticipator. 

Within  these  limitations,  however,  there  are  caseo  in  which  system  Inversion  is 
important.  For  example,  a  two-terminal  network  might  be  designated  as  a  system  with 
current  input  and  voltage  output,  and  we  might  want  to  determine  its  expression  with 
voltage  input  snd  current  output.  Or,  we  may  have  a  communication  channel  in  which 
the  effect  of  some  system  component  is  to  be  canceled  by  Introducing  anothei  component 
in  cascade. 

The  problem  of  determining  the  inverse  of  an  analytic  system  is  quite  aimllsr  to  the 
feedback  problem.  Designating  the  inverse  of  H  by  K,  we  have  the  system  equation 
H  •  ii  ■  ]•  As  in  the  feedback  problem,  an  easy  solution  is  possible  only  tf  Hq  ■  0,  and 
If  we  then  choose,  out  of  ihe  many  possible  solutions  for  KQ,  the  solution  K0  ■  0,  which 
will  always  exist.  We  then  obtain 

K,(a)  -  1/H  j(a) 


K2fs,,s2l  - 


-H2(s(,s2)  K,(s,>  K,(s2) 
H,(Sj  ♦  s2) 

-h2(»i,«2) 

H, <*,♦*,,>  H,(s,)  H,ls,) 


(91) 


and  further  system  transforms  can  be  calculated  successively.  It  can  be  verified  that 
these  terms  also  vatlsty  the  equation  &  •  H  ■  L  and  it  may  be  surmised  that  this  equa¬ 
tion  Is  satisfied  in  general,  so  that  is  a  true  inverse  of  i|,  but  a  general  proof  has 
not  been  fcumi. 

*  \e  procedure  for  proving  £  analytic  by  determining  bounds  on  Bg(x)  snd  Rg  ia 
similar  to  the  procedure  for  feedback  problems.  We  separate  if  into  a  linear  part  ii' 
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and  a  pari  H*  with  no  linear  term.  Then 


lh'  ♦  H*>  •  K  •  1  mi 

The  tnverae  of  H*  t»  K',  the  linear  approximation  to  K.  Cascading  each  side  of  Kq.  91 
with  we  have 

k*  •  (a*  ♦  a*i  *  ts  •  k* 

a.'  •  H '  *  K  ♦  K*  •  H*  *  l(  • 

a « a1  -  a*  ♦  a*  •  a  oji 

and  then  proceed  aa  In  section  4.  5. 

We  conclude  that  if  the  linear  approximation  to  the  inverse  of  an  analytic  system  haa 
an  absolutely  lntegrable  impulse  response,  then  the  inverse  Is  analytic. 

The  same  argument  shows  also  that  the  inverse  is  physically  realisable  if  lla  linear 
approximation  is  physically  realisable.  We  have  in  fact  obtained  K  ui  the  form  shown 
in  Fig.  5.  which,  if  II  and  K*  are  physically  realisable,  describes  a  phyalcal  real  its. 
tion  of  K  in  terms  of  physically  realizable  components.  * 

4.8  MEASUREMENT  O f  NONLINEAR  SYSTEMS 

It  has  been  shown  that  any  sy  stem  that  is  connnuous(R)  can  he  approximated  In  ana. 
lytic  form,  with  error  that  is  uniformly  smaller  than  any  prcaaslgned  tolerance.  The 
proof  waa  purely  an  existence  proof  that  gave  no  indication  of  a  method  for  obtaining 
an  approximation.  Although  several  methods  can  be  used  lo  obtain  analytic  approxlma. 
bone  to  given  systems,  none  of  these  methods  can  be  uaed  in  such  a  way  (hat  a  given 
tolerance  will  be  guaranteed. 

The  problem  is  similar  to  that  of  determining  polynomial  approximations  to  a  real 
function.  Three  methods  are  available;  determination  of  a  polynomial  that  equals  ths 
given  function  at  selected  points,  as  in  me  method  of  finite  differences;  expansion  In  a 
Taylor  scries;  and  expansion  in  a  series  of  orthogonal  polynomials.  Alt  these  methods 
can  be  generalized  and  used  for  systems  The  first  will  be  described  in  this  section, 
the  second  in  the  section  4.9,  and  the  third  will  be  discussed  in  Section  V. 

Consider  the  time. invariant  system  U  as  represented  by  the  functional  fj.  which 
gives  the  value  of  the  output  at  any  time  in  terms  of  the  past  of  the  input.  The  finite 
difference  of  £  with  reaped  to  the  real  function  ♦  is  defined  as 

■  h(u  ♦  e»  -  &<u)  (941 

This  defines  as  a  functional,  since  ti  specifies  s  real  number  for  every  real 
function  u  for  which  h(u)  and  h(u  ♦  e)  exiet.  We  can  then  conaider  the  system  A^H. 
which  ie  represented  by  the  functional  A^b,  »•  the  finite  difference  of  0  with 
respect  to  9. 
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Finite  difference*  can  be  taken  successively,  with  respect  to  the  same  or  different 
♦  -function*: 

A .  .  b(u)  ■  A  A  b(u>  •  A  h(u ♦  ♦.)  -  A.  h(u) 

’n  **  *1  *t  £  *t 

«  &(u  4j)  -  ll(u  ♦  ♦j)  -  lj(u  ♦  tz)  ♦  Wu>  • 

-  h(u  ♦  ♦,  ♦  42)  -  h(u  ♦  t  43)  -  h(u  ♦  4Z  ♦  43> 

♦  U<u  +  ♦,)  ♦  fe(u  ♦  4Z)  ♦  fc(u  ♦  43) 


The  general  form  for  finite  uiffc  re-  ces  of  any  order  can  be  Inferred.  These  forms  can 
be  used  for  the  experimental  determination  of  finite  differences. 

If  H  is  an  analytic  system,  its  finite  differences  will  also  be  analytic.  A  Straight* 
forward  calculation,  in  which  terms  of  like  degree  arc  collected,  gives  the  system 
functions  of  the  first  finite  difference,  if  we  assume  that  the  system  functions  hn  srs 
symmetric,  for  the  first  few  terms: 

>  ♦<*,)  #{r2)  «S-r1dT2  ♦  ... 

W° "  V''  V  «v  dTj 

♦(».)  4<t21  dr^Tj  ♦  ... 

A^h2(T|,Tz)  «  if  hjitj.Tj.  Tj)  4<t3)  d»3  ♦  ...  {9b) 

Mute  that  ho  doc*  not  appear  anywhere  in  these  equations,  and  that,  in  general,  h 
appears  only  in  A.h  ,,  A  .h  and  so  forth.  Therefore,  it  il  Is  *  sy  ■  em  of  degree 

iv»  *  n- 1  t  n-4 

the  N  finite  difference  will  involve  only  h^,  and  this  will  appear  only  in  the  constant 
term,  ao  that  the  Ntl1  finite  difference  is  a  constant-output  system: 

*“j[I  VT» . *H*  fi,  ♦t<Ti,-Tr*-d,N  <97> 

Thu"  an  experimental  determination  of  A^  _  _  ^tl(O)  for  all  possible  combliatlons 
of  .  ea.  '  ■  rtions  4^.  is  sufficient  to  determine  the  highest  order  system  function 

hjj.  When  this  has  been  determined,  the  undetermined  part  of  the  system  is  a  system 
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of  degree  K-  I,  and  therefore  all  of  the  system  function.  ;.tr  oe  determined. 

In  tact,  it  ta  not  necessary  to  use  combinations  of  all  possible  ^-functions.  If,  for 
example,  A^  _  ^tdOl  ia  determined  for  all  poaaible  combination*  of  functions 
chosen  from  tome  complete  normal  orthogonal  sequence,  then  we  obtain  the  coefficients 
of  an  expansion  of  hy  in  *  series  of  products  of  these  functions. 

If  we  use  impulses  as  the  s-funcuons,  designating  by  t  a  unit  impulse  occurring  st 
e  «  x,  then  we  have 

At  h(0)  «  N!  hv(x. . x.,)  (98) 

“i  *N  '  w 

If  we  use  step  functions,  designating  by  *X(T)  the  value  of  a  function  that  hag  value 
tcro  for  t  >  x,  and  value  1  foi  t  1  x,  then  we  have 

/*’•  r n 

ASX  . . .  S^hfO)  «  N1  J  ...  J  hN<Tl . TN)de,..fdeN  (9°) 

and  h^  can  be  deternun.  jy  uiffet entiating  th  a  ■ 

If  wc  use  expontntiais,  with  tin  notation  e^tri  «  e”*  ',  then 

Aes1--  eaNb,0)!*NI  HKUi . *K>  ,100) 

An  equivalent  result  can  be  obtained  by  using  Unusoida  instead  of  exponentials. 

If  we  have  a  system  that  it  not  analy  tic  u.'.:  .  i.:e  jegree,  we  can  assume  that  it  is 

approximate  by  an  analytic  system  of  degree  S  and  apply  the  procedure  suggested.  We 
then  derive  an  approximate  analytic  system  that  gives  the  same  output  as  the  given  sys¬ 
tem  for  the  inputs  that  were  used  to  determine  the  finite  d.rfe.-ences. 

The  response  of  a  linear  system  to  any  input  can  be  determined  if  we  know  its 
response  to  a  unit  impulse,  to  a  unit  step,  oi  to  all  of  the  sinusoids  of  a  given  amplitude. 
We  now  have  reason  to  believe  that  this  can  be  extended  to  nonlinear  systems  in  the  fol¬ 
lowing  way.  Thr  response  of  a  continuous  nonlineir  system  to  any  input  can  be  deter, 
mined,  at  least  approximately,  iS  we  know  ita  response  to  all  possible  combinations  of 
unit  impulses,  to  all  possible  combinations  of  unit  steps,  or  to  all  possible  combinations 
of  sinusoids  of  all  frequencies  with  a  given  amplitude, 

4.9  TAYLOR -SERIES  EXPANSIONS  OF  ANALYTIC  SYSTEMS 

Through  the  theory  of  funetio"  x  of  a  real  or  a  complex  variable  we  have  come  to 
associate  analyticity  with  differ*. .liability,  as  well  as  with  representation  in  a  power 
aeries.  The  differentiation  of  an  analytic  system  is  not  only  of  mathematical  interest 
(cf.  Hillc  (14)],  but  also  useful  in  the  determination  of  the  system  functions  of  an  ana¬ 
lytic  system  by  a  Taylor-aeries  method. 


47 


As  in  vector  analysis,  for  example,  when  the  tlirectlonni  derivative  of  a  scalar  func - 
ttnn  of  position  depends  upon  the  choice  of  a  direction  in  space,  an  analytic  system  does 
not  have  a  unique  derivative.  We  shall  define  the  derivative  of  an  analytic  system  in 
terms  of  the  functional  that  represents  it,  and  we  shall  define  it  as  essentially  a  direc . 
tioual  derivative. 

The  derivative  of  the  functional  1)  with  respect  to  the  real  function  4  if  defined  as 
the  limit  of  a  finite  difference  quotient: 

b<u*«e)  -  b(u) 

h  (u)  ■  lim - - -  1101) 

♦  t-0  * 

’’his  derivative  h^  is  also  a  functional.  The  time-invariant  system  that  It  defines  will 
be  called  1]^,  the  derivative  of  H  with  respect  to  4.  We  can  now  differentiate  h^  with 

respect  to  a  function  4,  and  obtain  the  second  derivative  h  ,,  and  successive  derivatives 

— w 

can  be  defined  ad  infinitum. 

If  Ij  is  analytic,  and  its  system  functions  are  assumed  to  be  symmetric,  then  a 
Straightforward  calculation  of  the  derivative  shows  that  the  derivative  is  analytic,  with 
system  functions 


,n(Ti- 


•V 


(nil) 


/' 


hntl*V 


nf  I 


*V»' 


(1011 


If  4  is  in  PB1(M).  for  any  M  (that  is  not  necessarily  less  than  ftH),  then 


ll%,J  *  (ntl)  M|hn+|p 


<101) 


from  which  it  follows  that  the  bound  function  of  l]_  is  no  greater  than  M  times  the  derlv- 

4 

alive  of  the  bound  function  of  H,  and  its  radius  of  convergence  is  not  less  than  R^. 

It  follows  from  the  assumed  symmetry  of  the  system  functions  hn  that  the  higher 
order  derivatives  with  respect  to  different  functions  are  independent  of  '.he  order  of  dif¬ 
ferentiation.  Since,  however,  a  system  is  not  changed  by  making  its  system  functions 
symmetric,  this  conclusion  is  true  whether  the  derivative  is  calculated  from  symmetric 
or  unsymmetric  system  functions. 

Applying  the  . ot  inula  (Eq,  102)  for  the  derivative  n  limes,  we  find  that  the  constant 
term  of  the  nth  derivative  is 


r,.. 


■V 


1  n! 


/:•••/: 


hJT. 
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Thus.  hn  can  be  determined  from  the  values,  for  u  »  0,  of  the  11th  derivatives  of  H.  The 
samp  comment  made  in  the  discussion  of  finitr  dtffcrtnce*  applies  here  also.  The 
4-functions  for  which  t.n  derivatives  must  be  determined  need  not  be  ail  possible  func  ¬ 
tions,  and  we  can  use  in. pulses,  step  functions,  or  sinusoids,  as  in  section  4.8.  This 
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mull  can  bt  uaed  «i  the  bun  at  a  method  tor  the  nuu 

Unear  system*.  It  u  not  hard  to  due  tot 


hO  a  o  '  ,im  *♦ 

♦r’Vs  *♦{•••**. 


ao  that,  for  the  application  of  the  Taylor-nene*  method,  *e  can  calculate  the  higher 
order  differences  aa  limits  of  (imte-duflereace  quotient*.  computed  for  «  •  0,  instead 
of  by  computing  successive  derivatives  for  all  a. 

The  expression  for  the  system  function*  in  terms  of  derivatives  provides  a  proof  of 
the  statement  that  the  symmetric  forms  o<  the  system  functions  arc  unique. 

The  T»)lor-series  method  can  be  used  only  for  analytic  systems,  since  it  dentil 
the  system  functions  from  the  small-signal  response  of  the  system,  and  therefore  deter¬ 
mines  the  large-signal  response  from  the  small -signal  response  by  extrapolation.  Its 
sdvantage  over  the  method  of  finite  differences  is  that  it  is  not  restricted  to  systems  of 
finite  degree.  As  an  experimental  method,  it  has  the  disadvantage  that  it  inwo'ves  limits 
of  observed  values,  so  that  many  observations  may  be  necessary  to  obtain  e  -*  limit, 
while  the  method  n f  (mite  differences  requires  only  the  observation  of  output  valves  for 
specific  inputs. 

4. 10  SYNTHESIS  OF  ANALYTIC  SYSTEMS 

Two  methods  will  be  mentioned  for  the  synthesis  of  systems  from  their  analytic 
representations.  The  first,  obviously  appticah’ '  only  in  special  cases,  consists  in 
recognizing  already  familiar  forms  ir  the  sy*  .cm  functions.  It  is  Illustrated  in  sec¬ 
tion  4.6.  in  whicn  the  second-degree  spproxun..con  to  the  solution  of  >  feedback  problem 
was  synthesised  in  cascade  term.  Continued  investigation  of  the  properties  and  nppU. 
ratloiis  of  analytic  representations  can  be  expected  to  make  Uus  method  applicable  lo 
an  ever-widening  class  of  systems. 

The  second  method  is  based  on  the  fact  that  if 


hn(Tl . v*nw  .  -  <,0*‘ 

then  this  term  of  the  system  can  be  synthesised  as  a  product  of  linear  systems,  if  hB  la 
a  sum  of  such  products,  then  It  can  be  synthesized  as  a  sum  of  products  of  linear  sys¬ 
tems.  Such  a  product  expression  can  be  obtained  by  expanding  bn  m  a  series  of  products 
of  orthogonal  functions.  However,  no  investigation  has  been  made  for  determining  the 
conditions  under  which  this  method  can  be  expected  to  vivid  arbitrarily  small  error. 

Neither  these  methods  nor  the  methods  proposed  by  Wiener  and  Bose  can  be  consid¬ 
ered  satisfactory  methods  of  synthesis.  All  of  these  methods  are  capable  of  the  following 
absurdity:  we  are  given  a  black  box,  we  examine  it  In  the  laboratory  by  applying  mforts 
and  observing  the  output*,  we  assemble  several  hundred  dollars'  worth  of  equipment  tc 
approximate  its  characterised,  and  then  open  the  box  and  find  thst  it  con  is. ms  a  handful 
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V.  OKTKOOOXAU  SYSTEMS 


i.  I  MEASURE  AND  INTEGRAL 

The  purpose  of  nui  teciun  u  to  attorn,  in  a  general  eve  ***  the  (ipuin^A  of  real 
functions  in  senes  of  orthogonal  fmcuons  can  be  rttiribf  to  mdueate  a  method  for 
expanding  nonlinear  systems  in  senes  of  orthogonal  systems.  Sewe  the  expansion  of 
real  functions  tr*  this  method  involves  the  process  of  eompccmg  the  integral  of  a  real 
function,  the  application  of  the  method  to  systems  will  require  integration  of  systems, 
or  at  leas!  .cffi  taon  of  fi-ictionals.  The  idea  of  Integra  taco  mast  therefore  be  extended 
from  real  fain  -  s  to  systems  or  fiascuooals. 

Fur  th.-  -  w  it  is  appropriate  to  indicate  here  sax  of  the  fundamental  ideas  of 
integration,  in  this  section  the  reiauoo  between  integrauae  usd  measure  theory  will  be 
roughly  outlined,  and  it  will  be  ihosa  that  the  theory  of  probability  plays  an  important 
part  in  the  ir.icg* .tia*  cf  • » 1 , ^ ti '  ~  r . 

We  begin  with  the  integration  of  real  functions.  Suppose  we  bare  a  real  f-aicuun  f, 
and  want  to  integrate  it  over  the  intern!  (A,  B),  The  integral  that  se  obtain  is  defined 
in  geometric  terms  as  the  net  area  I  v tween  the  s.axis.  the  graph  of  y  »  f(x).  ajd  the 
>ines  i  i  A,  x  a  B.  This  area  is  obta-ned,  in  principle,  as  me  1.  mil  ci  a  sc ,t-u - -•  •  tJ 
approximations,  each  approxi mauoa  being  obtained  by  d:-»disg  toe  area  i  :«  -  .an 
of  strips  and  estimating  the  width  and  i‘e  length  of  each  cnp.  In  ge  ■  ..  -*•  fj-i 

are  parallel  to  the  >  -axis. 

In  the  classical  definition  of  the  Riemann  integral  .  ttrtjs  are  constructed  t, 
dividing  the  interval  (A.  B)  into  a  Urge  number  of  small  rtesertili  and  taking  u  . 
aubintervaj  as  defunct  width  of  a  strip.  The  length  of  ec  rfc  strip  is  calimaied  as 
some  •  alue  ass.  *ned  S  fi,|  on  the  subinlrrvai  that  defines  its  width.  The  integral  is 
estimated  as  the  s  the  areas  of  the  strips,  and  the  integral  is  is  lined  as  die  Limit 

of  these  estimates  .  ,  at  interval  (A,  B)  is  divided  into  smaller  and  smaller  subintervala. 
If  f  is  continuous,  the  range  over  wh.ch  ffx)  varies  on  each  Mtwnterval  becomes  srbi. 
trarily  small  as  the  subintervals  are  made  small,  and  the  estimate  of  the  length  of  the 
strip  becomes  better  and  better,  so  that  the  limit  that  defines  the  integral  exists.  But 
if  f  is  not  continuous,  flx)  may'contmue  to  vary  widely  no  matter  how  small  the  aubln. 
terval  is  made,  and  the  integral  thus  defined  will  not  ezi*:. 

To  get  around  thia  difficulty,  we  tedeflne  the  amps  in  such  a  way  that  fla)  cm  not 
.ary  widely.  -We  4.  <nde  the  y -axis,  instead  of  the  x -axis,  into  subintervals,  so  that  the 
length  of  each  strip  can  be  estimated  with  error  that  la  leas  tau  the  width  of  the  subin* 
terval.  and  define  Use  width  of  the  strip  as  the  total  length  of  thr  set  of  all  a  .‘or  which 
f(x)  is  in  the  submierval.  If  f  is  continuous,  this  set  cf  x’a  si1  consist  of  a  collection 
of  discrete  intervals,  and  the  total  l*ogih  of  this  set  is  simper  the  sum  of  the  lengths  of 
their  intervals.  But  if  f  is  not  continuous,  this  set  may  m t  be  a  collection  of  intervals, 
and  we  are  fa.ed  with  the  problem  of  defining  the  total  l«— ig-N  of  a  set  of  nun  hers  th-:  is 
not  an  interval  and  cannot  be  decomposed  into  intervals. 
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The  Lebesgue  theory  of  mniure,  by  providing  at  lc.\st  a  partial  aolution  to  this  prob¬ 
lem,  achieves  the  desired  generalisation  of  integration.  This  theory  shows  how  we  can 
assign,  to  each  of  a  large  collection  of  aeta  called  measurable  sets,  a  number  called  Its 
measure,  so  that  the  measure  of  a  set  has  the  properties  that  we  associate  with  the  idea 
of  total  length:  the  measure  of  a  set  is  always  non. negative,  and  the  measure  of  the 
umo>:  of  a  finite  or  countable  collection  of  nonoverlapping  seta  is  the  sum  of  the  measures 
of  the  component  sets.  In  particular,  the  measure  called  Lebesgue  measure  has  tbs 
property  that  the  Lebesgue  measure  of  an  interval  is  the  same  as  ita  length.  Dy  means 
of  Lebesgue  measure  we  can  proceed  to  define  integration  for  many  functions  that  art 
not  continuous:  for  each  subinterval  of  the  range,  we  multiply  the  measure  of  the  cor¬ 
responding  subset  of  the  domain  by  some  value  lr  the  subinterval  of  the  range,  take  the 
sum  of  these  products  as  an  estimate  of  the  integral,  and  define  the  limit  of  these  esti¬ 
mates,  as  the  subintervale  are  made  amaller,  as  the  Lebesgue  integral  of  the  function. 

The  point  in  which  we  are  Interested  here  is  not  that  the  Lebesgue  integral  la  defined 
foi  functions  whoae  Kiemann  integral  is  not  defined,  but  rather  that  the  theory  of  inte¬ 
gration  in  terms  of  measure  can  be  used  to  define  Integrals  of  functions  that  are  not  real 
functions.  Consider  any  real-valued  function,  whose  domain  may  be  a  set  of  any  kind  of 
objects.  Divide  the  range  of  the  function  into  sublntervais.  and  foi  each  subuiterval  con¬ 
sider  the  subset  or  the  domain  on  which  me  value  of  the  function  lies  in  that  aubinterval. 

If,  to  .very  such  subset  of  the  domain,  we  can  assign  a  number  that  wa  can  call  its  meas¬ 
ure,  then  we  can  proceed,  just  as  in  th.  case  of  the  Lebesgue  Integral,  to  define  the  inte¬ 
gral  of  the  function. 

The  proLlem  now  is  (odd  net  measur  an  a  set  of  real  functions,  and  the  solution 
comes  from  the  theory  of  pn  babiitty.  A  probability  ensemble  le  a  set  c.*  objects,  In 
which  wc  assign  to  every  subset  —  or  at  least  to  certain  subsets  —  a  number  called  the 
probability  of  that  subset.  The  probability  of  the  whole  ensemble  is  unity;  the  probability 
of  every  subset  is  non-negative,  and  does  not  exceed  unity,  and,  in  fact,  probability  has 
all  the  properties  that  arc  required  of  s  measure. 

Therefore,  if  we  have  a  real-valued  function  whose  domain  is  a  probability  ensemble, 
we  can  define  an  integral  of  that  function.  We  divide  the  range  of  the  function  into  sub¬ 
intervals,  and  fot  each  subinterval  we  multiply  some  value  in  that  subimerval  by  the 
probability  that  the  value  of  the  function  will  lie  in  that  aubinterval.  We  add  these  prod¬ 
ucts  ever  all  subintervaib  to  obtain  an  estimate  of  the  integral.  It  will  be  seen  that  by 
this  process  we  have  obtained  an  estimate  of  th#  ensemble  average,  or  expectation,  of 
the  value  of  the  function.  An  integral,  defined  in  terms  of  probability  measure.  Is  simply 
an  ensemble  average.  „ 

To  define  the  integral  of  a  functional,  all  we  need  is  an  ensemble  of  real  (unctions  a. 
The  integral  of  the  functional  Is  then  the  ensemble  average  of  the  value  of  the  functional 
for  this  ensembls  of  u.  The  integral  of  a  time-invariant  system  can  then  be  defined  if 
we  have  a  stationary  ensemble  of  inputs  f.  The  stattonariness  of  the  enaemble  implies 
that  the  enecrr.bls  of  functions  u{,  u{(t)  ■  f(t-r),  will  be  the  same  for  every  t,  and  the 


integral  of  ilw  system  is  defined  an  the  Integral,  for  this  eneemble  of  u.  of  the  functional 
that  rtpretmu  it;  that  it,  the  eeteemUe  average  of  the  value  of  the  output  of  the  a  yet  eta. 
In  ovary  practical  situations,  the  statmnary  ensemble  of  inf  ate  U  preacrihed  by  the  appii- 
Ckiim  for  which  the  system  la  be  mg  considered. 

i.2  EXPANSIONS  IN  ORTHOGOVU.  SYSTEMS 

Every  *  '  *»- invariant  eyatea  can  be  repreaented  by  a  functional,  and  every  stationary 
ensemble  of  function*  f  correspond*  to  some  ensemble  of  functions  u.  which  la  the  aaae 
at  toe  eneemble  of  functions  u(  for  any  t.  Therefore,  we  shall  do  all  our  mathematical 
work  us  terms  of  functionals;  and  the  results  can  be  translated  immediately  in  terms  of 
system*  by  the  fact  that  the  average  of  the  value  of  a  functional  will  equal  the  average  of 
the  value  of  the  output  of  the  aystem  that  it  represents. 

Suppose  we  have  available  in  the  laboratory  a  bank  of  nonlinear  time. Invariant  «;>• 
tern*  and  suppose  we  also  have  an  unknown  time. invariant  aystem  U  that  is  to  be 
approximated  as  a  linear  combination  of  the  systems  Representing  these  as  fuoc- 
utaula,  we  *UaU  determine  real  numbers  c(  with  the  property  that  the  functional 
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is  aa  approximation  to  t-  The  error  of  approximation  will  be  the  output  of  the  aystem 
I»B*  H*.  represented  by  the  faartkaaal 

N 

K  -  tl  -  Jh*  l,0«> 

i«l 


We  shall  obtain  an  approximate  representation  of  U  In  terms  at  the  systems  <Jj, 
This  representation  may  be  useful  ia  two  distinct  ways.  If  th*  systems  ^  are  easy  to 
construct,  then  we  have  a  way  of  constructing  H  or  an  approximation  to  it.  If  the  sys¬ 
tems  ^  have  convenient  mathematical  representation*,  we  obtain  a  convenient  mathe¬ 
matical  representation  of  ft. 

Suppose  that  the  criterion  of  approximation  is  that  th*  mean. square  value  of  the 
error,  for  a  particular  stationary  ensemble  of  inputs,  be  a*  small  a*  possible.  We  shall 
designate  the  mean  or  expectation  of  the  value  of  a  functional  by  an  irterral  sign,  to 
show  that  our  mathematics  ia  analogous  lo  the  mathematic*  of  real  function*.  Than  ww 
moat  determine  the  numbers  so  that 
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M  a  minimum.  At  the  minimum  nlat,  me  partial  derivative* 
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must  all  be  aero.  Tbe  condition  for  minimum  mean-square  error  is  therefore  that,  for 
every  k. 
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It  is  possible  to  Implement  this  scheme  in  the  laboratory.  The  output  of  each  of  the 
systems  is  passed  through  an  adjustable -gain  atnp'lficr,  which  provides  for  the  adjust¬ 
ment  of  c{.  and  the  outputs  of  these  amplifiers  are  added  to  construct  the  system  U*. 

The  output  of  l|«  is  subtracted  from  tbe  output  of  H  to  obtain  the  error  aystero  E.  Now 
we  must  Multiply  the  output  of  each  systems  by  the  output  of  E  and  obtain  the 

ensemble  average  of  this  product,  li  toe  ensemble  of  tnp  its  is  trgodic,  this  ensembls 
average  will  equal  a  time  average,  a  (as  Wisr.cr  has  suggested)  we  might  combine  the 
multiplication  with  the  estimation  of  tbe  tune  average  by  using  an  overdamped  electro- 
dynamometer.  Then  we  have  a  but  of  N  meters,  and  we  must  adjust  the  N  gain  con. 
trot*  so  that  each  meter  is  made  to  read  aero. 

In  the  general  case  this  may  be  a  troublesome  procedure,  because  the  adjustment  of 
a  Single  gam  control  may  change  tbe  readings  of  all  the  meters.  However,  If  the  ays. 
terns  9)  *re  *u<h  that 
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so  that  the  reading  of  the  k1**  meter  is  affected  only  by  the  tt*  gain  control,  and  the 
appropriate  adjustment  can  be  made  quite  simply,  Furthermore,  if  the  meters  and  the 
gam  controls  are  appropriately  calibrated,  we  can  perform  the  adjustment  by  setting 
each  gain  control  lo  aero,  reeding  each  meter,  and  then  setting 
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The  combtlM  imposed  oa  the  lyiimi  ^  can  be  expressed  in  a  terminology  that  Is 
conventional  (or  the  analogous  situation  in  the  theory  ot  real  (unction*.  We  shall  say 
that  the  system*  are  all  normaUzed,  and  that  all  are  orthogonal  to  each  other.  The 
set  of  systems  will  be  called  an  orthonormal,  or  normal  orthogonal,  set.  The  hpproxt- 
mation  obtained  by  this  procedure  will  be  called  an  expansion  in  orthogonal  systems. 

Since  these  conditions  were  imposed  in  connection  with  s  particular  ensemble  ot  inputs, 
we  shall  speak  of  systems  normal  and  orthogonal  with  respect  to  s  particular  input 
ensemble. 

If  the  systems  ss  gives,  are  not  normal  and  orthogonal  with  respect  to  the  par¬ 
ticular  input  ensemble  that  we  intend  to  use,  we  can  consu  ct,  by  means  of  a  well-known 
procedure,  a  set  of  linear  combinations  of  'hem  that  are  normal  and  orthogonal.  This 
procedure  can  be  described  by  supposing  that  we  have  already  constructed  s  set  of  n 
normal  and  orthogonal  systems,  and  we  have  a  system  that  is  not  normal  or  orthog¬ 
onal  to  these  n  system*.  We  are  to  construct  a  linear  combination  of  these  n  ♦  I  sys¬ 
tems  which  is  normal  and  orthogonal  to  the  first  n.  We  do  this  by  constructing,  with 
the  use  of  the  first  n  systems,  s  minimum-mean. square- 1  rror  approximation  to  9n+i* 
The  system  whose  output  is  the  error  of  this  approximate ,«  Is  ortliog'nai  to  the  first  n 
systems,  and  if  it  is  not  equivalent  to  zero,  then  we  can  normalise  it  by  multiplying  It 
by  an  appropriate  constant.  (If  it  is  equivalent  to  xero,  then  every  linear  combination 
of  the  n  ♦  I  systems  can  be  also  obtained  with  the  first  n.  and  the  additional  system 
is  of  no  us  to  >in .) 

In  genera*.  . -n.  all  we  need  to  obtain  minimum -mean-square-error  approximations 
is  a  bank  of  no>in«ear  systems,  some  adjustable -gain  amplifiers,  and  some  product- 
average  meters.  The  given  systems  can  be  orthoqonalised  and  ran  then  be  used  to  obtain 
orthogonal  expansions  of  any  given  system. 

How  close  can  these  approximations  be  made 7  Can  the  mean. square  error  of  approx- 
imation  be  made  arbitrarily  small  by  using  a  large  enough  bank  of  nonlinear  systems  g^V 
For  systems  that  sre  conUsuous(H),  considered  only  for  ensembles  of  Inputs  that  are 
bounded(ft),  we  have  a  ready  answer.  We  begin  with  s  sequence  of  linear  system* 
with  impulse  response  function*  kj,  rich  that  there  i*  no  function  u  not  equivalent  to 
xero  for  which 


s: 


u(t)  k4(r)  dr 


Ot*) 


tor  all  i.  (The  sequence  of  system*  whose  impulse  response*  are  the  La  guerre  func¬ 
tions  has  this  property.)  Then  we  form  a  sequence  of  non'lnc  >  system#  consisting  of 
a  constant-output  system,  these  linear  systems,  and  »U  pro'll  .1  of  combination*  of 
these  systems.  The  set  of  *11  linear  combinations  of  these  . .  sr  systems  is  aa 
algebra  that  separates  points  (cf.  section  2,)),  sod  therefore  any  system  that  ts  contln- 
uout(R)  can  be  approximated  arbitrarily  closely  by  such  Untar  combinations,  in  tho 
sense  that  for  any  positive  number  <  there  exist*  a  linear  combination  of  these  systems 


»» 
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whose  output  never  differ*  from  the  output  of  the  given  system  by  more  than  (.  It  follows 
immediately  that  the  mean-square  error  can  thus  be  made  lets  than  <*.  The  method  of 
orthogonal  expansions  will  yisld  approximation*  with  mean-square  error  that  Is  ss  small 
aa  can  be  obtained,  so  that,  by  using  a  sufficiently  large  number  of  systems,  the  mean- 
square  error  can  be  made  arbitrarily  email. 

This  does  not  imply  that  the  method  of  orthogonal  expansion  can  be  made  to  yield 
approximations  with  uniformly  small  error.  However,  il  is  easily  seen  that  the  proba¬ 
bility  of  an  error  of  magnitude  greater  thar  any  number  A  cannot  exceed  (c/A)  if  the 
mran-Kfar  t  error  is  ti  or  smaller,  so  that  we  have  (for  the  ensemble  with  respect  to 
which  the  expansion  was  made)  an  almost  un.formly  small  error  in  a  statistical  sense. 

An  nju.Miv n  made  with  respect  to  one  input  ensemble  will  have  email  error,  although 
not  minimum  error,  when  some  other  input  ensembles  are  used.  Consider  two  ensembles 
of  inputs.  Eg  and  E2,  consisVng  of  the  same  set  of  inputs  with  different  probability  dis¬ 
tributions.  Suppose  ihat  the  system  H  has  been  approximated  by  means  of  an  orthogonal 
expansion  with  respect  to  Ej.  so  thet  (if  we  attach  the  ensemble  designation  to  the  inte¬ 
gral  sign)  we  have 


If  the  probability  of  any  subset  in  ensemble  Ej  ia  never  greater  than  M  times  ihe  proba¬ 
bility  of  the  same  subset  in  E,,  then  obviously 


Then  by  making  the  mean-square  error  that  is  measured  with  Ej  sufficiently  small,  we 
can  make  the  mean-square  error  that  is  measured  with  E2  *•  small  as  may  be  re  ;•  -red. 
Thus  a  small  mean-square  error  in  one  ensemble  implies  a  small  mean-square  error 
in  another  ensemble.  Furthermore,  even  under  the  looser  condition  that  every  set  with 
xero  probability  in  E(  must  have  zero  probability  In  E2,  we  can  conclude  that  with  input 
ensemble  E2  there  must  be  nntnU  probability  of  Urge  error,  even  though  no  b-wind  on  the 
mean-,  qua  re  error  can  be  set.  There  is  reason  to  believe  that,  if  the  system  that  la  to 
be  approximated  is  continuous,  even  this  condition  car.  be  relaxed,  and  that  there  may 
be  orthogonal  methods  of  deriving  approximations  with  uniformly  small  error,  (It  is 
known  that  such  methods  exist  for  the  approximation  of  real  functions.) 

Some  of  the  preceding  discussion  is  applicable  to  the  expansion,  in  orthogonal  sys¬ 
tems,  of  systems  that  are  not  ccntinuouc  (for  example,  hysteretic  systems).  In  this 
case,  if  the  nonlinear  lystem*  ^  are  continuous  (as  they  will  be  if  they  are  products  of 
linear  system*  with  Laguerre-function  Impulse  responses),  an  approximation  with 
uniformly  small  error  cannot  be  obuined.  However,  It  may  be  poeeible  to  obtain 


i 


approximations  «i6  ntll  mean-square  error,  perhaps  even  arbitrarily  small.  At  any 
rate,  the  smallest  mean-square  er-or  that  is  possible  with  a  continuous  approximation 
can  be  approached. 

5.  S  l  LLTERIKG  AND  PREDICTION 

The  same  method  can  also  be  used  In  minim  urn -me  an -square -error  (ilia  r  in  |  and 
prediction  problems,  if  the  joint  ensemble  of  Input  and  desired  output  can  be  produced 
in  the  laboratory.  Vs  proceed  precisely  as  though  the  desired  output  had  been  obtained 
as  the  output  of  a  system  to  which  the  Input  was  applied.  The  results  of  the  preceding 
section  then  Imply  that,  with  any  bank  of  nonlinear  filters  ^  that  ts  adequate  to  approxi¬ 
mate  a  continuous  system  *vtth  arbitrarily  small  mean-square  error,  are  can  come  arbi¬ 
trarily  close  to  any  pin.  .'intna  that  -an  be  achisveq  with  a  continuous  filter. 

Nott  that  there  assy  be  some  problems,  no  optimum  continuous  filter.  If  the  opti¬ 
mum  filler  Is  ik  ■mr'us.  (A  heuristic  lll'ttlr.'liw  is  (<  untl  In  the  theory  of  real 
functions:  the  ..  *>»tn“'  y  tn  I 
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